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SECTION  I 


INTRODUCTION 


This  report  J  j  intended  to  support  the  long-term  objective 
of  digital  signal  synchronization  within  the  Defense  Communica¬ 
tions  System  (DCS).  The  work  described  in  this  document  covers 
a  3-1/2  year  program  consisting  of  design,  development,  fabrica¬ 
tion,  test  and  evaluation  of  three  Timing  Subsystem  (TS)  proto¬ 
types.  The  results  obtained  and  recommendations  presented  should 
provide  insight  to  the  future  advanced  development  model  of  a 
DCS  timing  subsystem  which  will  distribute  coordinated  timing 
signals  at  each  node. 

The  planned  DCS  digital  transmission  network  can  be  visual¬ 
ized  in  its  most  elementary  form  as  a  collection  of  nodes  inter¬ 
connected  by  radio  links  or  cascades  of  links.  Bit  streams  ori¬ 
ginating  from  geographically-separated  sources  enter  the  node 
and  typically  require  multiplexing  for  retransmission  to  a  com¬ 
mon  destination  node.  However,  the  loss  of  proper  timing  in  a 
synchronous  digital  system  is  catastrophic.  When  bits  do  not 
fall  in  their  assigned  time  slots,  all  received  information  is 
meaningless  --  a  totally  unacceptable  situation.  This  charac¬ 
teristic  makes  the  tiraing/synchronization  function  one  of  the 
most  important  functions  in  a  digital  communications  system 
[1.1].  Problems  have  occurred  in  the  timing  relationship 
between  different  digital  communications  networks  that  were  not 
originally  engineered  to  communicate  with  one  another.  In  some 
cases,  these  problems  were  overcome  by  the  addition  of  variable 
storage  buffers  and  adequate  clocks  to  control  them,  while  in 
other  cases  modification  of  equipments  might  also  be  required. 
Even  after  these  corrective  measures,  it  might  be  necessary  to 
occasionally  interrupt  traffic  to  reset  the  variable  storage 
buffers.  It  is  not  always  possible  at  the  time  of  equipment 
development  to  predict  just  what  system  interfaces  will  be 
required  during  the  lifetime  of  an  equipment.  Therefore,  design 
of  the  equipment  to  meet  a  minimum  timing  compatibility  standard 
is  highly  desirable  for  avoiding  future  problems.  Such  a 
timing  compatibility  standard  should  follow  a  well-developed 
DCS  timing  plan.  See  [1.1]. 

In  the  past  few  years,  various  methods  of  avoiding  or 
minimizing  these  undesirable  phenomena  have  been  proposed,  in¬ 
cluding  independent  highly  stable  atomic  clocks  [1.2],  mutual 
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frequency  averaging  [1.3],  hierarchical  master/slave  [1.4],  and 
self-organizing  master/slave  [1.5].  More  recently,  considera¬ 
tion  has  been  given  to  the  use  of  network  facilities  for  system- 
wide  transfer  of  a  time  reference  [1.6]  [1.7],  and  theoretical 
models  have  been  used  to  predict  relationships  between  time 
transfer  accuracy  and  link  parameters.  To  satisfy  normal  com¬ 
munication  requirements,  relative  time  synchronization  of  the 
nodes  is  sufficient,  i.e.,  the  node  clocks  need  not  be  phased 
identically  as  long  as  their  mutual  average  frequency  offsets 
are  zero.  On  the  other  hand,  transfer  of  a  time  reference 
throughout  a  network  is  equivalent  to  the  requirement  that  node 
clocks  be  synchronized  with  zero  phase  offset.  The  additional 
hardware  required  for  time  reference  transfer  does  not  appear 
to  be  significant  for  a  production  timing  subsystem  model,  and 
the  benefits  of  a  systemwide  time  reference  certainly  justify 
further  examination. 

The  DCS  network  involves  a  large  number  of  links  and  nodes 
with  various  categories  of  transmission  media,  including  line- 
of-sight  microwave  (LOS),  troposcatter  (TROPO) ,  satellite,  and 
cable.  The  variety  of  transmission  equipment  that  is  available 
now,  or  planned  as  part  of  the  all-digital  network,  makes  a 
complete  and  comprehensive  evaluation  of  system  performance 
difficult;  therefore,  an  emphasis  in  the  experimental  work 
reported  on  here  has  been  toward  timing  technique  synchroniza¬ 
tion  performance  on  a  small  (three  node)  Test  Bed. 

This  effort,  under  contract  with  CNR,  Inc.,  Needham,  MA, 
was  conceived  to  produce  three  Timing  Subsystem  prototypes  in 
order  to  provide  the  necessary  measurement  and  processing 
capabilities  to  implement  and  evaluate  a  series  of  timing 
technique  candidates .  Although  many  techniques  for  timing  and 
synchronization  have  been  proposed  and  studied,  as  yet  a  direct 
comparison  in  the  field  had  not  been  attempted.  Ihe  modes  of 
operation  are  Independent  Clocks,  Mutual  Synchronization, 
Master/Slave,  Time  Reference  Distribution,  and  Improved  Time 
Reference  Distribution.  The  candidates  were  implemented  in 
nodal  software.  Laboratory  as  well  as  field  evaluation  deter¬ 
mined  the  capabilities  and  performance  characteristics  of  each 
method.  Valuable  insight  was  gained  concerning  the  implemen¬ 
tation  difficulty  or  other  pecularities  associated  with  the  tim¬ 
ing  technique  candidates.  Perhaps  overlooked  as  a  simple  task, 
the  practice  of  keeping  all  the  link  equipment  operational 
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proved  to  be  far  more  difficult  than  originally  anticipated. 
Setup  and  shakedown  time  as  well  as  equipment  maintenance 
efforts  were  compounded  due  in  part  to  the  distance  between 
sites.  Node-to-node  communication  was  through  TROPO  and  LOS 
links . 

Three  Timing  Subsystem  prototypes  were  constructed  to 
enable  network-level  timing  and  synchronization  experiments  on 
the  RADC  (Rome,  NY)  Test  Bed.  The  results  of  this  com¬ 
parison  are  presented  in  this  report.  Also  included  are  a 
brief  description  of  the  Timing  Subsystem  hardware  and  software 
components  as  illustrated  in  Section  2.  This  section  labels  the 
major  functional  portions  of  the  equipment  as  well  as  the 
methodology  behind  the  design  and  implementation  of  nodal  soft¬ 
ware  to  control  and  orchestrate  processing,  link  input/output 
and  performance  monitoring  tasks.  Although  not  actually  part 
of  the  Timing  Subsystem,  the  role  of  the  microcomputer  develop¬ 
ment  system  as  both  a  software  development  facility  and  a  net¬ 
work  controller  is  also  discussed  in  this  section. 

Section  3  labels  the  criteria  and  follows  the  series  of 
events  leading  to  the  final  network  configuration  used  for 
both  two-node  and  three-node  timing  experiments.  Detailed 
descriptions  of  link  equipment  connections  to  the  timing  sub¬ 
system  enhance  the  presentation  of  the  tandem  network  radio 
links.  Next,  an  account  of  node  equipment  connections  brings 
to  light  the  scheme  used  to  evaluate  clock  performance  during 
timing  technique  tests.  Finally,  a  choice  of  Test  Bed  master 
clock  is  made  and  selection  criteria  reviewed. 

The  first  equipment  delay  measurements  are  revealed  in 
Section  4  which  is  devoted  to  itemizing  modem,  radio  and  path 
delays  of  link  equipment  used  for  network  synchronization  tests. 
A  look  at  the  procedure  used  for  measuring  equipment  delay 
automatically  through  the  Timing  Subsystem's  back-to-back  error 
measurement  facilities  precedes  tabularized  results  for  both 
LOS  and  TROPO  paths.  Where  precise  measurements  were  not 
possible,  findings  from  a  previous  CNR,  Inc.  effort  [1.8] 
supplement  the  data.  Section  4  closes  with  a  "roadmap"  summary 
of  all  the  one-way  delay  elements  of  the  tandem  network. 

A  review  of  the  candidate  timing  techniques  and  how  each 
fits  into  a  general  classification  of  a  time /frequency  syn¬ 
chronization  hierarchy,  sets  the  framework  for  the  ensuing 
description  of  both  laboratory  and  field  tests  presented  in 


Section  5.  Included  in  this  chapter  is  a  discussion  of  the 
methodology  behind  the  performance  measurement  system  using 
both  the  Timing  Subsystem’s  and  Frequency  Measurement  Terminal's 
(FMT)  [1.9]  time  interval  measurement  capabilities.  This  serves 
to  clarify  the  presentation  of  results  of  the  network  synchroni¬ 
zation  tests.  Section  5  closes  with  a  summary  of  the  experi¬ 
ment  program  and  node  parameters  for  each  test.  A  review  of  the 
time  transfer  mechanism  and  error  computation  techniques  is 
also  included. 

Section  6  contains  recommendations  for  a  future  timing 
subsystem  advanced  development  model  as  well  as  interim  subsys¬ 
tem  (AN/GSQ-183)  interfacing  and  operation  when  used  as  a 
backup  frequency  source.  Section  6  continues  by  discussing 
network  timing  accuracy  considerations,  timing  technique  candi¬ 
dates  and  general  recommendations  for  future  work  in  network 
synchronization.  Appendix  A  contains  summaries  of  the  soft¬ 
ware  implementations  of  Time  Reference  Distribution  [1.6]  and 
Improved  Time  Reference  Distribution  [1.7]  as  interpreted  by 
CNR,  Inc. 
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SECTION  2 


EQUIPMENT  DESCRIPTION 


Three  prototype  Timing  Subsystems  were  tested  in  a  small 
network  TEST  Bed  at  RADC.  The  Timing  Subsystems  (TS’s)  are  res¬ 
ponsible  for  clock  data  transmissions  between  nodes  and  the  up¬ 
dating  of  local  references.  Between  no^ss  there  is  an  exchange 
of  clock  control  data,  as  shown  by  the  "Time  Reference  Infor¬ 
mation  Packets"  (TRIP's).  These  data  packets  spurt  between 
all  connected  nodes  at  a  rate  of  4  kb/s  beginning  at  every 
second.  Their  primary  function  is  the  transfer  of  clock  control 
data  and  node  reference  information. 

Network  control  is  accomplished  by  telephone  access  to  each 
of  the  TS’s  through  the  couplers  and  modems  provided  by  a  Fre¬ 
quency  Measurement  Terminal  (FMT)  located  at  each  site.  It  is 
through  these  means  that  each  Timing  Subsystem  and  FMT  may  be 
interrogated  and  experimental  results  obtained. 

2 . 1  Timing  Subsystem 

The  basic  function  of  a  Timing  Subsystem  is  to  combine 
locally-made  timing  measurements  with  timing  information  trans¬ 
mitted  from  other  nodes  in  order  to  compute  the  local  node 
clock  error  relative  to  a  master  reference.  The  nature  of  this 
reference  is  determined  by  the  particular  synchronization 
technique  being  evaluated.  Then,  based  on  the  computed  clock 
error,  the  TS  corrects  its  local  node  clock  and  transmits  local 
timing  information  to  other  nodes. 

Figure  2.1  shows  a  photograph  of  the  three  Timing  Subsys¬ 
tems  and  the  Network  Controller.  Each  subsystem  consists  of 
a  double  bay  chassis  with  a  total  rack  mountable  equipment 
space  of  84  inches.  Following  is  a  list  of  the  removable  assem¬ 
blies  and  the  rack  space  each  occupies: 

Left  Side 

•  Power  Supply  Drawer  (10") 

•  Spare  (16") 

•  Card  Cage  (16") 
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Right  Side 

•  Display  Panel  (10") 

•  Frequency  Distribution  Unit  (5") 

•  Spare  (5") 

•  Phase  Microstepper  (3") 

•  Spare  (3") 

•  Distribution  Amplifier  (3") 

•  Spare  (3") 

•  Spare  (10") 

Figure  2.2  shows  a  block  diagram  of  a  Timing  Subsystem. 

The  locally-made  timing  measurements  are  performed  by  the  Time 
Interval  Measurement  unit,  and  timing  information  from  other 
nodes  is  extracted  from  the  Receive  Link  Termination  Processors 
(RCV  LTP's).  The  node  control  computer  computes  the  local  node 
clock  error  and  corrects  the  clock  (node  frequency  standard) 
with  a  software  controlled  phase  microstepper  (or  VCO) .  Local 
timing  information  is  transmitted  to  the  nodes  via  the  Transmit 
Link  Termination  Processors  (XMIT  LTP's). 

2.1.1  Node  Control  Computer 

The  Timing  Subsystem  (TS)  computer  is  microprocessor-based 
equipment  designed  around  the  Motorola  6800  series  of  devices. 
Control  and  computational  hardware  are  located  on  a  single 
wire-wrap  board  while  I/O  devices  are  split  between  two  addi¬ 
tional  boards.  Flexibility  of  the  TS  is  enhanced  by  the  exten¬ 
sive  use  of  programmable  I/O  for  controlling  external  equipment. 

In  this  subsection,  we  briefly  describe  each  of  the  major 
hardware  building  blocks  incorporated  within  the  node  computer. 
Figure  2.3  shows  a  functional  block  diagram  of  the  node  control 
computer  giving  an  indication  of  the  breakdown  of  processing 
functions.  Notice  the  Interrupt  Dispatcher  is  the  routine  that 
orchestrates  software  execution  with  the  TS  programs. 

The  Node  Control  Processor  (NCP)  consists  of  the  following 
major  elements,  each  of  which  is  described  briefly  in  this 
subsection.  A  more  complete  hardware  description  is  contained 
in  [2.1]  : 
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Figure  2.3  Node  Control  Computer  Functional  Block  Diagram 


•  Microprocessor 

•  Random  Access  Memory  (RAM) 

•  Read  Only  Memory  (ROM) 

•  Arithmetic  Processing  Unit  (APU) 

•  Serial/Parallel  1/0 

•  Link  Termination  I/O 

•  Interrupt  Priority  Encoder 

The  heart  of  the  Timing  Subsystem  is  the  Motorola  M6800 
microprocessor.  When  executing  the  node  control  programs,  the 
M6800  is  the  controlling  device  within  the  equipment.  Through 
the  programmable  capabilities  of  many  external  I/O  devices,  the 
microprocessor  can  transmit  and  receive  information,  read  status, 
control  external  devices,  and  supervise  node  computational  pro¬ 
cessing.  The  following  list  describes  some  of  the  features 
of  this  device.  A  more  detailed  description  may  be  found  in  [2.2). 

Some  of  the  more  important  features  of  the  MC6800  micro¬ 
processing  unit  that  contribute  to  the  "ease  of  use"  in  a  system 
are : 

•  8-bit  bidirectional  data  bus 

•  16-bit  address  bus  -  65K  bytes  of  addressing 

•  72  instructions  -  variable  length 

•  Seven  addressing  modes  (direct,  relative,  immedi¬ 
ate,  indexed,  extended,  implied,  and  accumulator) 

•  Interrupt  vectoring 

•  Two  accumulators 

•  Index  register 

•  Program  counter 

•  Stack  pointer  and  variable  length  stack 

•  Condition  code  register  (6  codes) 

•  Separate  nonmaskable  interrupt 

Briefly,  computer  memory  in  the  Timing  Subsystem  is  divided 
into  two  sections:  Random  Access  Memory  (RAM)  and  Read  Only 
Memory  (ROM) . 
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The  M6800  microprocessor  has  the  capability  to  address  up 
to  65K  (216)  locations  from  its  16-bit  address  bus.  The  Timing 
Subsystem  has  8K  (2^3)  bytes  of  RAM  storage  available  for  use 
by  the  microprocessor  for  scratchpad,  parameter  storage,  point¬ 
ers,  buffers,  etc.  This  is  volatile  semiconductor  memory  that 
does  not  save  bits  when  power  is  removed.  Below  is  a  list  of 
some  of  the  more  important  functions  RAM  is  used  for: 

•  Process  scheduling  tables 

•  Programmable  parameter  storage 

•  XMIT,  RCV  buffers 

•  XMIT,  RCV  pointers 

•  System  monitor  pointers  and  scratchpad 

•  PLL  and  arithmetic  scratchpad 

•  Performance  assessment  data  storage 

•  Diagnostic  buffers  and  pointers 

•  System  stack 

Node  control  programs  reside  in  ROM  located  on  the  CPU 
board.  Vectoring  to  the  entry  point  of  the  system  programs  is 
automatic  upon  application  of  AC  power.  Program  development 
was  made  easier  with  a  microcomputer  development  system  used 
during  the  program.  The  development  station  also  has  a  ROM 
programmer  to  load  and  bum  ROMs  for  installation  in  the  Timing 
Subsystem.  Included  in  the  development  system's  capabilities 
is  a  hardware  emulator  that  was  used  to  debug  the  Timing  Sub¬ 
system  computer  sections. 

The  Arithmetic  Processing  Unit  (APU)  is  a  monolithic  device 
that  provides  high  performance  fixed  and  floating  point  arith¬ 
metic  and  a  variety  of  floating  point  trigonometric  and  mathe¬ 
matical  operations. 

All  transfers,  including  operand,  result,  status,  and 
command  information,  take  place  over  the  8-bit  system  data  bus. 
Operands  are  pushed  onto  an  internal  stack  and  a  command  is 
issued  to  perform  operations  on  the  data  in  the  stack.  Results 
are  then  available  to  be  retrieved  from  the  stack,  or  additional 
commands  may  be  entered.  Reference  [2.3]  further  describes  this 
device. 
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Transfers  to  and  from  the  APU  are  handled  directly  by  the 
microprocessor.  Upon  issuing  a  command  to  the  APU,  the  device 
is  polled  by  the  CPU  until  APU  execution  is  completed.  This 
method  of  control  is  perhaps  not  the  most  efficient  use  of 
processor  time;  however,  it  greatly  simplifies  program  control 
and  software  overhead.  Additionally,  worst-case  APU  computation 
times  are  not  slow  enough  to  require  use  of  Direct  Memory  Access 
(DMA)  or  APU  interrupt  capabilities. 

2.1.2  Link  Termination 


The  Timing  Subsystem  computer  hardware  contains  a  signifi¬ 
cant  number  of  I/O  devices  to  facilitate  external  communication 
with  the  node.  Summarized  below  are  each  type  of  I/O  hardware 
and  its  associated  functions.  Figures  2.4  and  2.5  depict  all 
devices  under  direct  microprocessor  control. 

Parallel  I/O  is  handled  by  Peripheral  Interface  Adapters 
(PIA)  under  microprocessor  control.  Each  PIA  supports  the 
following  functions: 

•  Two  3-bit  bidirectional  data  buses  for  interface 
to  peripherals 

•  Two  programmable  control  registers 

•  Two  programmable  data  direction  registers 

•  Four  individually-controlled  interrupt  input  lines; 
two  usable  as  peripheral  control  outputs 

•  Handshake  control  logic  for  input  and  output 
peripheral  operation 

These  devices  are  interfaced  with  various  external  hardware 
and  allow  direct  microprocessor  control  of  the  following 
functions: 

•  Commands  and  status  for  front  panel  display  for 
all  six  links 

•  Microstepper  control  and  data  interface 

•  Time  interval  measurement  unit  address  selection 
and  data  interface 

•  All  other  front  panel  display  indicators 

•  VCO  option  data  port 

•  Time-of-day  (TOD)  clock  interface 
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OUTGOING  INCOMING 


Figure  2.5  Node  Control  Processor 


The  Link  Termination  Processors  (LTP's)  support  the  clock 
data  communications  through  the  service  channel  multiplexers. 

In  particular,  they  support  the  Synchronous  Data  Link  Control 
(SDLC)  protocol  with  the  following  functions: 

•  Zero  insertion  and  deletion  for  runs  of  l's 

•  FLAG,  ABORT,  and  GO  AHEAD  bit  sequences 

•  Frame  check  sequence  generation  and  transmission 
(CRC-CCITT,  x16  +  X12  +  X5  +  1) 

•  Error  checking  at  receiving  station 

•  Secondary  station  address  comparison  with  frame 
address  (optional) 

Clock  data  is  transmitted  in  an  SDLC  information  frame 
(TRIP).  The  frame  contains  timing  information  necessary  for 
clock  correction. 

These  devices  service  the  complex  protocol  required  for 
transmit  and  receive  data  patterns.  There  are  12  ADLC  devices 
in  the  TS  hardware. 

A  third  type  of  I/O  device  under  computer  control  is  the 
Asynchronous  Communication  Interface  Adapter  (ACIA) .  These  pro¬ 
vide  a  convenient  RS-232  standard  interface  for  the  operator's 
console.  It  is  used  along  with  a  programmable  dual  baud  rate 
generator.  Significant  features  include  the  following: 

•  8-  and  9-bit  transmission 

•  Optional  even  and  odd  parity 

•  Parity,  over-run,  and  framing  error  checking 

•  Programmable  control  register 

•  Up  to  500-kb/s  transmission 

•  Double-buffering 

The  clock  data  is  transmitted  in  an  SDLC  information 
frame  (TRIP).  It  is  used  to  send  initialization  data  around 
the  network,  and  also  contains  timing  and  status  information 
necessary  for  clock  correction. 
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The  data  format  for  a  TRIP  is  shown  in  Figure  2.6.  It 
contains  all  of  the  clock  update  parameters  for  the  link  in 
question,  as  well  as  performance  assessment  parameters.  The 
clock  update  parameters  include  the  following: 

Node  Status  -  sync  and  overflow  condition  flags  for 
radios  and  buffers;  also,  loop  mode,  e.g.,  acquisi¬ 
tion  and  tracking. 

Transmit  Time  (t^)  -  measured  time  of  previous  out¬ 
going  TRp  signal  relative  to  1  pps . 

Receive  Time  (t_)  -  measured  time  of  previous  incoming 
TRP  signal  relative  to  1  pps. 

Clock  Error  (u^)  -  current  estimate  of  clock  error. 

TRP  Rank  and  Merit  Parameters  -  TRD  decision  para¬ 
meters  ,  N2,  N^,  N^. 

Second  Count  -  time  in  seconds  at  last  1-pps  tic  at 
transmit  node  (stored  in  a  shifted  binary  pattern). 

The  number  of  bits  allowed  for  each  of  these  parameter  fields 
is  seen  near  the  bottom  of  Figure  2.6,  along  with  the  standard 
SDLC  control  address,  flag,  and  frame  check  fields. 

The  remaining  packet  capacity  is  allocated  to  performance 
assessment  parameters.  In  addition,  provision  has  been  made  for 
transmission  of  date  and  time  of  day  resolved  down  to  1  minute; 
this  is  transmitted  only  on  the  minute  boundaries  in  order  to 
conserve  space  in  the  1-second  data  transmissions.  The  perfor¬ 
mance  assessment  parameters  include  the  following: 

Current  Reference/References  in  Use  -  For  TRD,  the 
node  currently  being  referenced  by  the  transmit 
node;  for  mutual  sync,  the  references  used  in  the 
formation  of  the  clock  correction  signal. 

Buffer  Fullness  -  The  relative  fullness  of  each  of  the 
six  FIFO's  at  the  transmit  node. 


Absolute  Clock  Error  -  Time  difference  between  TEST 
BED  master  1  pps  and  local  node  reference  1  pps. 


Finally,  it  should  be  noted  that  time -of -day  variables  are 
transmitted  as  8-bit  ASCII  (for  a  total  of  96  bits)  when  the  time 
marker  message  is  transmitted  (once  per  second) .  Other  para¬ 
meters  are  single  bit  flags,  addresses  (8-bit  bytes),  or  floating 
point  variables  (24-bit  mantissa  plus  7-bit  exponent  plus  a  sign 
bit).  The  total  bit  count  comes  to  536  bits,  which  sets  the 
minimum  data  rate  requirement  at  536  b/s. 

Figure  2.7  shows  a  functional  block  diagram  of  a  transmit 
LTP.  It  communicates  with  the  node  control  computer  through  a 
data  bus  interface.  Data  bytes  to  be  sent  out  through  the 
LTP  are  passed  from  the  data  bus  interface  to  a  FIFO,  and  then 
converted  to  serial  form  by  the  shift  register.  The  cyclic 
reducdancy  error  check  sequence  is  added  at  this  point,  and  the 
data  then  passes  through  a  zero  insertion  circuit.  This  cir¬ 
cuit  insures  that  no  data  byte  will  look  like  a  flag  byte.  The 
output  data  is  scanned  by  the  flag  detect  circuit  to  indicate 
the  transmission  of  a  flag,  which  is  used  as  a  TRP  for  Option  C 
[2.4]. 

Figure  2.8  shows  a  functional  block  diagram  of  a  receive 
LTP.  Incoming  data  passes  through  a  zero  deletion  circuit  (per¬ 
forms  complementary  function  of  XMIT  LTP  zero  insertion  circuit) 
to  a  serial  in,  parallel  out  shift  register.  The  data  bytes 
then  are  fed  to  a  FIFO  and  get  to  the  node  control  computer 
through  a  data  bus  interface.  The  incoming  data  has  the  cyclic 
redundancy  error  check  sequence  verified;  errors  are  indicated 
in  a  status  register  which  can  be  read  by  the  node  control 
computer.  The  flag  detect  circuit  provides  the  TRP  output  to 
the  time  interval  measurement  unit  for  Option  C. 

All  of  the  functions  of  a  link  termination  processor  are 
implemented  with  a  single  LSI  integrated  circuit  -  Motorola's 
Advanced  Data  Link  Controller  (MC6854) .  This  circuit  supports 
a  1  Mb/s  data  rate,  bit-oriented  protocols  (SDLC,  ADCCP, 

HDLC),  error  checking  (CRC16),  and  has  an  easy  interface  to 
the  6800  microprocessor. 

Due  to  the  extensive  use  of  interrupts  within  the  Timing 
Subsystem,  a  custom  interrupt  encoder  that  essentially  priori¬ 
tizes  24  levels  of  hardware  interrupts  handles  the  interrupt 
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Figure  2.8  Receive  Link  Termination  Processor  (RCV  LTP)  Block  Diagram 


routing  chores.  Each  of  the  incoming  interrupts  is  ANDed  with 
a  programmable  mask  register  producing  an  enabled  and  disabled 
interrupt  request  line.  A  holding  register  latches  each  request 
line  and  guarantees  synchronism  with  the  microprocessor  states 
by  using  the  same  process  clock  as  the  MPU.  Next,  as  hardware 
interrupts  are  received,  a  cascaded  priority  encoder  arrange¬ 
ment  allows  interrupt  service  based  on  hardware  priority.  Ihe 
software  interrupt  vector  is  generated  by  a  set  of  ROMs  with 
each  interrupt  having  a  unique  jump  table  location.  Control  is 
transferred  to  the  location  pointed  to  by  the  address  found  in 
the  jump  table. 

2.1.3  Time  Interval  Measurement 

The  basic  measurement  capability  cf  the  Timing  Subsystem 
is  the  recording  of  time  intervals  between  once-per-second 
pulses  via  the  Time  Interval  Measurement  Unit.  This  single 
board  interval  counter  contains  14  read-only  registers  and  one 
write-only  select  register.  The  node  computer  first  selects 
a  channel  and  then  reads  a  number  showing  elapsed  time  from  the 
source  (adjustable  1-pps  output  from  the  Timing  Subsystem)  to 
the  pulse  from  the  selected  channel.  Measurements  are  latched 
in  the  output  register  indefinitely  depending  on  the  state  of 
the  select  word  and  in  the  internal  registers  for  a  period  of 
one  second. 

Each  of  the  Time  Reference  Pulses  (TRPs)  has  a  period 
of  one  second,  so  the  maximum  measurement  interval  is  one  sec¬ 
ond.  The  system  has  10-ns  resolution  because  the  measurement 
is  made  with  counters  clocked  at  a  100-MHz  rate.  The  number  of 
bits  used  to  describe  the  time  interval  at  the  100-MHz  rate 
is  1  s/10  ns  *  10®  and  <  10®  *  2^ .  Therefore,  a  27-bit 
counter  is  used. 

There  may  be  up  to  15  incoming  time  interval  measurements 
each  second  to  be  recorded  for  subsequent  transfer  to  the  node 
control  computer.  This  is  done  with  10  27-bit  registers 
whose  outputs  are  multiplexed  to  the  computer.  The  operation  is 
as  follows:  The  local  1-pps  reference  resets  the  counter  which 
then  proceeds  to  count  10-ns  time  intervals.  Upon  arrival  of  a 
time  reference  pulse,  the  counter  contents  are  strobed  into 
one  of  the  registers,  and  an  interrupt  is  issued  to  the  computer. 
The  interrupt  service  routine  now  outputs  the  multiplexer  select 
signals  and  reads  in  the  measurement  data. 
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The  six  outgoing  time  interval  measurements  are  under  con¬ 
trol  of  the  transmit  Link  Termination  Processors.  Therefore, 
the  measurements  can  be  done  sequentially,  using  only  one  27-bit 
storage  register.  Operation  goes  as  follows:  The  local  l-pps 
reference  resets  the  counter  and  issues  an  interrupt  to  the 
computer.  The  computer  responds  by  initiating  a  transmission 
through  one  of  the  LTP's.  The  TRP  from  this  LTP  strobes  the 
counter  contents  into  the  register.  The  computer  now  reads 
the  measurement  data  from  the  register  and  initiates  the  next 
LTP  transmission.  This  continues  until  all  six  transmit  LTP 
transmissions  are  complete. 

Figure  2.9  shows  a  block  diagram  of  the  TIM  board. 

2.1.4  Node  Frequency  Standards 

The  choice  of  frequency  source  for  the  Timing  Subsystem 
station  standard  centers  around  cost  and  site  equipment  avail¬ 
ability.  If  at  a  given  site,  a  frequency  standard  is  not 
available,  the  Timing  Subsystem  must  employ  its  own  internal 
frequency  source  to  allow  stand-alone  operation.  During  the 
design  phase  of  the  program,  it  was  anticipated  that,  in  cer¬ 
tain  tests,  the  Timing  Subsystems  would  be  equipped  with  atomic 
standards.  Therefore,  space  was  provided  in  the  chassis  for  such 
units.  However,  two  of  the  three  TS  prototypes  also  contained 
Voltage  Controlled  Oscillators  (VCO)  to  enable  stand-alone  opera¬ 
tion  for  performance  comparisons  using  more  stable  oscillators 
[2.5].  This  device  was  the  station  standard  whenever  an  atomic 
standard  was  not  available.  Under  these  circumstances,  its 
output  was  used  to  drive  the  station  synthesizer  and  frequency 
distribution  system.  The  characteristics  of  the  quartz  oscil¬ 
lator  are  quite  satisfactory  for  the  intended  applications 
where  the  Timing  Subsystem  is  operated  in  disciplined  modes. 
Furthermore,  the  high  servo  update  rate  of  1  per  second  is 
well-matched  to  the  use  of  quartz  frequency  sources  in  the 
subsystems. 

Whether  from  an  external  source  or  an  internal  oscillator, 
the  TS  requires  a  5-MHz  frequency  source  to  be  used  as  station 
standard  time.  Also,  a  method  to  adjust  that  source  based  on 
servo  loop  corrections  must  be  present.  The  two  alternatives 
are  a  voltage  controlled  oscillator  or  a  phase  microstepper. 

The  former  is  controlled  by  the  node  computer  via  a  D/A  inter¬ 
face  while  the  latter  is  manipulation  directly  by  the  computer 
through  a  parallel  interface. 
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The  inherent  stability,  both  short  term  and  long  term, 
of  the  frequency  standard  plays  only  a  minor  role  for  the  case 
of  a  slaved  clock  but  a  much  more  important  one  for  the  case 
of  a  network  master  source  (undisciplined).  Therefore,  a  dis¬ 
ciplined  clock  need  not  be  of  highest  quality  (i.e.,  quartz).  A 
direct  comparison  of  frequency  sources  may  be  found  in  Table  2-1. 

2.2  Timing  Subsystem  Software  Development 

During  the  early  stages  of  the  program,  it  was  proposed 
to  design  a  rather  general  purpose  microcomputer  capability 
into  the  Timing  Subsystem  hardware.  It  was  essential  that  a 
combination  of  minimal  complexity,  low  cost  and  ease  of  pro¬ 
gramming  be  incorporated  in  such  a  design.  This  most  certainly 
ruled  out  an  external  general-purpose  microcomputer  as  a  TS 
controller  because  it  involved  more  hardware  and  software 
capability  than  was  needed  plus  the  obvious  factor  of  increased 
cost.  Clearly,  a  custom  designed  but  general-purpose  "micro¬ 
computer  on  a  board"  was  the  most  cost  effective  approach. 

With  extensive  use  of  programmable  I/O  hardware  and  a  floating 
point  arithmetic  device,  the  Timing  Subsystem  was  given  the 
powerful  data  handling  and  computational  capabilities  required 
to  terminate  up  to  6  links,  provide  error  processing  for  each 
as  well  as  general  nodal  timing  tasks.  Design  effort  was  mini¬ 
mal  and  followed  standard  microprocessor  applications  suggest 
tions.  The  previous  subsection  describes  in  further  detail 
the  computer-based  hardware. 

Timing  Subsystem  software  was  a  natural  candidate  for 
modular  construction.  Considering  that  most  processing  func¬ 
tions  are  triggered  by  external  events  (TRP  pulses,  1-pps 
events,  etc.)  modularity  became  a  necessity.  Software  tasks 
were  broken  down  into  functional  pieces  and  coding  performed 
after  detailed  flowchart  analysis.  The  orchestration  of  many 
external  interrupts  and  the  relative  asynchronism  between  them 
required  the  use  of  a  process  arbitration  function  to  be 
implemented  in  software.  This  process,  called  the  Interrupt 
Dispatcher,  is  responsible  for  scheduling  events  and  arbitrating 
between  routines  requesting  service.  In  the  special  case  when 
there  are  no  scheduled  processes,  the  Interrupt  Dispatcher 
will  transfer  control  to  the  system  monitor  software  to  detect 
any  pending  commands.  Thus,  the  Timing  Subsystem  software 
allows  interactive  operation  with  a  user  even  when  the  system 
is  fully  loaded.  Maximum  flexibility  is  achieved  by  incorporat¬ 
ing  this  small  monitor  as  a  part  of  the  nodal  software.  Most 
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commands  are  issued  with  a  single  keystroke.  Figure  2.10  de¬ 
picts  the  top  level  software  loop  just  mentioned. 

Once  system  operation  has  begun  via  keyboard  command,  all 
timing  functions  are  automatic  and  the  operator's  terminal  may 
be  used  as  an  interactive  monitor. 

Control  flow  for  the  software  modules  is  shown  in  Figure 
2.11.  Notice  that  several  processes  execute  immediately  with¬ 
out  notifying  the  Interrupt  Dispatcher.  These  routines  share 
in  common  highest  priority  and  minimal  execution  time,  thus 
eliminating  the  need  for  process  arbitration.  A  more  detailed 
software  description  may  be  found  in  [2.1]. 

To  encapsulate  the  philosophy  behind  Timing  Subsystem 
software  development,  it  is  important  to  realize  that  given  the 
hardware  to  perform  link  termination  time  interval  measurement 
and  frequency  distribution  tasks,  then  adding  microprocessor 
interfacing  for  control  of  this  equipment,  the  groundwork  for 
a  powerful  and  flexible  device  is  set.  By  using  the  same  hard¬ 
ware  and  merely  manipulating  software,  any  number  of  timing 
functions  or  system  configurations  may  be  implemented.  In  this 
light,  the  Timing  Subsystem  may  be  considered  a  fully- 
programmable  frequency  measurement  device  capable  of  automatic 
clock  disciplining. 

2.2.1  Microcomputer  Development  System 

Timing  Subsystem  software  development  was  made  less  cum¬ 
bersome  through  the  use  of  a  microcomputer  development  system. 
Apart  from  its  powerful  software  aids,  including  disk  operating 
system  and  file  management,  edit,  assembly  and  debug  programs, 
it  represents  a  general  microcomputer  product  easily  adapted 
to  many  applications  requiring  processing,  storage  and  dis¬ 
play  capabilities. 

The  development  systems  hardware  includes  a  dual  floppy 
disk  drive ,  keyboard,  CRT,  line  printer,  RS-232C  interface,  and 
in-circuit  emulator.  Also  built  in  is  an  IEEE-488  instrument 
interface.  The  system  is  pictured  in  Figure  2.12. 

Through  the  use  of  the  development  system,  editing, 
assembly  and  linkage  chores  were  handled  conveniently.  All 
software  was  coded  directly  in  assembly  language.  Instructions 
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(a)  Timing  Subsystem  with  Development  System 
and  PROM  Programmer 


(b)  Development  System  with  Disk  Drive  and  Printer 
Figure  2.12  Microcomputer  Development  System 


and  addressing  modes  are  that  of  the  Motorola  6800  micropro¬ 
cessor  family.  Included  with  the  development  system  was  an 
In-Circuit  Emulator  (ICE)  package  designed  with  an  emulator 
card  and  a  cable  attached  to  a  probe  which  plugs  in  directly  to 
the  socket  of  the  removed  microprocessor.  The  development 
station  may  be  instructed  to  emulate  the  6800  microprocessor 
functions.  Trace,  break,  single  step  and  the  ability  to  view 
the  internal  states  of  the  micro  further  simplified  debugging 
tasks . 

The  software  development  scheme  as  shown  in  Figure  2.13 
clarifies  the  technique  used  to  create  and  modify  programs. 

Once  modular  section  tasks  are  defined,  an  assembly  language 
source  file  (or  many  source  files)  is  assembled  and  linked  with 
other  program  sections .  These  sections  are  positioned  to  the 
memory  locations  actually  used  in  the  Timing  Subsystem.  Once 
loaded  into  the  microcomputer  memory,  the  binary  object  code  is 
copied  into  ROM  (actually  EPROM)  exactly  as  it  appears  in  the 
development  system.  Intermediate  debugging  was  aided  by  using 
the  emulator  feature  (as  described  earlier)  thus  avoiding 
repeated  ROM  erase-bum  cycles. 

2.2.2  Development  Station  as  Network  Controller 

The  network  controller's  role  is  to  set  up  an  experiment 
by  means  of  parameter  transfers  to  the  TS's,  and  then  become  a 
passive  monitor.  Resync  and  reorganization  tasks  are  handled 
exclusively  by  the  TS's.  The  controller/TS  interaction  is 
designed  so  that  sets  of  parameters  in  the  disk  files  may  be 
downloaded  to  the  remote  TS's,  thereby  avoiding  the  necessity 
of  travel  by  personnel  to  unattended  sites  merely  to  change  a 
switch  setting.  As  far  as  performance  monttoring  is  concerned, 
various  status,  measurement,  and  error  variables  will  be 
continually  broadcast  in  an  unreserved  portion  of  the  TRIP's 
and,  by  means  of  a  suitable  routing  scheme,  will  eventually  find 
their  way  to  the  network  controller. 

The  essence  of  the  approach,  then,  is  to  provide  a  central 
control  and  monitoring  function  in  the  network  to  set  up  and 
oversee  the  three  Timing  Subsystems.  This  unit  will  plug  into 
any  one  of  the  TS's  and  yet  be  capable  of  acquiring  data  or 
interrogating  any  of  the  remaining  TS's  which  will  normally 
be  at  remote  sites.  The  controller,  although  capable  of  stand¬ 
alone  operation  for  file  handling,  editing,  and  assembly,  will 
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be  considered  as  an  integral  part  of  the  TEST  Bed  network. 

Data  files  will  be  stored  on  the  disks  with  parameter  entries 
in  the  files  selected  to  represent  a  given  network  synchroni¬ 
zation  technique  and  TS  operational  mode.  Network  initializa¬ 
tion  parameters  may  be  taken  from  the  disk  file  and  trans¬ 
ferred  to  the  TS  via  the  RS-232C  control  port.  This  technique 
essentially  mimics  operator  keystrokes  but  does  so  at  a  much 
higher  rate. 

Performance  assessment  data  is  gathered  and  maintained  by 
each  TS  from  the  past  24  hours  of  operation.  Data  transfer 
from  the  node  may  be  requested  by  the  operator  through  a  printing 
terminal  or  the  development  station  connected  to  that  node. 

Although  a  network  management  scheme  was  not  implemented, 
provisions  for  incorporating  node  management  through  TRIP's 
already  exists.  Using  such  a  technique,  the  network  controller 
would  pass  setup  and  parameter  information  to  each  node  from  a 
central  location.  Each  TS  would  be  assigned  a  reference  number 
and  would  interpret  management  data  intended  only  for  the  local 
node.  Additionally,  performance  assessment  results  would  be 
sent  from  all  nodes  to  arrive  at  the  network  controller. 

For  this  program,  a  single  telephone  interface  was  set  up 
at  each  site  using  the  data  couplers  provided  by  the  Frequency 
Measurement  Terminals.  Nodal  management,  setup,  and  parameter 
transfers  were  accomplished  by  accessing  each  TS  individually. 
Through  this  technique  all  control  and  monitoring  functions 
were  handled  from  a  single  location.  This  was  appropriate 
for  a  smaller  network  but  perhaps  a  larger  network  would  bene¬ 
fit  from  a  node  management  protocol  gaining  TS  control  through 
TRIP  exchanges  sourced  from  a  central  location. 
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SECTION  3 


TEST  BED 


Originally,  four  Timing  Subsystem  prototypes  were  to  be 
built  and  tested  in  several  small  networks  representing  both 
loop  (delta)  and  tandem  configurations.  These  subsystems  would, 
ideally,  have  been  placed  at  four  distinct  sites  and  intercon¬ 
nected  with  at  least  one  redundant  link  to  allow  comparison  of 
clock  performance  under  realistic  conditions.  However,  in  the 
effort  of  minimizing  hardware  expenses,  it  was  agreed  that  only 
three  Timing  Subsystems  would  be  built.  This  limited  the  var¬ 
iety  of  network  configurations  possible  although  the  use  of 
both  TROPO  and  LOS  microwave  paths  to  support  a  timing  function 
could  still  be  accomplished. 

The  Time  Reference  Distribution  techniques  were  intended 
to  be  advantageous  in  a  larger  network  where  link  redundancy 
and  closed  loop  timing  paths  (contributing  to  potential  network 
instability)  are  likely  to  exist.  Clearly,  in  a  three-node 
network,  the  candidate  timing  techniques  would  be  expected  to 
behave  similarly  if  not  identically  insofar  as  our  measurement 
capabilities  could  detect  (with  the  exception  of  independent 
clocks).  With  these  thoughts  in  mind,  we  were  forced  to  dismiss 
the  idea  of  rigorously  testing  each  of  the  timing  technique 
candidates.  Instead,  we  emphasized  the  importance  of  developing 
a  Timing  Subsystem  capable  of  supporting  the  measurement  and 
link  termination  duties  required  as  well  as  possessing  a  great 
deal  of  flexibility  (due  largely  to  its  programmable  nature)  in 
the  field.  However,  even  limited  exposure  to  the  network 
synchronization  experiments  performed  provided  much  information 
and  helped  to  clarify  potential  problems  at  the  system  level. 

This  topic  is  explored  in  further  detail  in  Section  6. 

A  small  network  of  microwave  and  troposcatter  links  was 
configured  at  RADC,  Rome,  NY.  Here  we  briefly  review  the 
equipment  and  other  resources  available  at  the  site  installa¬ 
tions.  In  addition,  we  describe  the  particular  communication 
links  and  associated  equipment  used  during  the  field  test  phase 
of  this  program.  First,  we  offer  a  review  of  the  events  and 
selection  processes  that  led  to  the  final  network  used. 
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The  RADC  test  sites  include  both  line-of-sight  microwave 
and  troposcatter  facilities.  Enough  atomic  clocks,  digital 
modems,  radio  equipment  and  test  equipment  was  kept  at  each 
site  to  support  the  test  effort.  The  site  selection  process 
was  influenced  more  by  the  type  of  radio  link  (i.e.,  microwave, 
TROPO)  and  the  operational  status  of  communications  equipment 
than  by  the  availability  of  support  (i.e.,  test  instruments) 
gear.  However,  it  seemed  wise  to  include  both  LOS  microwave 
and  troposcatter  links  in  our  experiments  especially  since  a 
previous  study  by  CNR,  Inc.,  [3.1]  had  sufficiently  demonstrated 
that  supporting  a  timing  function  over  a  fading  channel  was 
feasible.  We  intended  using  a  TROPO  channel  for  an  intemode 
link.  At  the  time,  the  Verona  to  Youngstown  TROPO  path  was 
available.  This  link  was  supported  by  the  AN/TRC-132A  radio 
set  and  quad  diversity  DAR-IV  digital  modem.  Since  this  was 
the  only  TROPO  path  available,  a  tandem  network  was  the  only 
choice  for  network  configuration. 

The  selection  of  microwave  links  involved  a  few  more  fac- 
tors.  Originally,  two  Timing  Subsystems  were  installed  at 
Griffiss  AFB  to  allow  the  configuration  of  a  loop  or  delta 
network  from  Griffiss  to  Stockbridge  to  Verona  to  Griffiss. 

In  fact,  this  proposed  network  also  Included  an  up-down  satellite 
link  between  the  nodes  at  GAFB.  However,  the  satellite  radio 
set  had  suffered  a  blown  klystron  shortly  before  our  arrival 
and  a  new  tube  would  take  months  to  be  delivered.  The  satellite 
link  was  subsequently  scratched  from  the  test  plan.  After 
inspecting  site  equipment  and  reviewing  operational  status  of 
radios  and  modems,  It  was  determined  that  RADC  facilities  could 
provide  two  microwave  links.  The  first  link  was  an  8  kHz  LOS 
routed  between  Verona  and  GAFB  (Building  3)  through  a  repeat 
at  Stockbridge.  The  second  was  a  direct  14  GHz  link  between 
Verona  and  GAFB  (Building  3)  using  Terracora  TCM-608B  commercial 
equipment.  This  more  recently  installed  wideband  radio  set  was 
not  operational  at  experiment  time  so  in  order  to  test  the 
Timing  Subsystems  in  a  tandem  network,  one  of  the  two  TS's  at 
GAFB  was  shipped  to  the  Youngstown  site.  No  timing  experiments 
were  performed  with  a  delta  style  network.  This  decision  was 
made  with  respect  to  the  following  factors : 

•  The  satellite  link  would  not  be  available  for  use 
during  the  experimental  tests.  This  eliminated  the 
possibility  of  forming  a  delta  network  with  radio 
links.  Although  a  direct  wire  connection  could  be 


3-2 


used  between  the  nodes  at  GAFB,  it  would  be  of  little 
value  since  the  satellite  link  total  transit  time 
would  be  on  the  order  of  250  ms  and  link  character¬ 
istics  would  be  of  great  interest  if  it  were  to  support 
a  timing  function. 

•  It  was  not  clear  when  the  TCM-6Q8B  radio  terminals 
would  become  operational.  Several  months  had  already 
elapsed  which  were  occupied  with  equipment  shake-  * 
down  and  delay  calibration  tests  (see  Section  4)  while 
the  faulty  equipment  was  serviced.  A  wait  and  see 
attitude  was  adopted  for  a  short  time,  however,  it  soon 
came  down  to  a  simple  choice.  In  the  effort  of  separat¬ 
ing  all  nodes  to  different  geographical  locations,  it 
was  imperative  that  one  of  the  subsystems  be  removed 
from  GAFB  and  installed  at  Youngstown  (at  one  end  of 
the  TROPO  link) .  This  required  a  commitment  to  abandon 
any  delta  network  testing.  Given  the  choice,  a  tandem 
network  was  preferred  so  a  subsystem  was  shipped  there. 
Additionally,  if  the  TCM-608B  radio  set  did  become 
operational,  it  could  be  used  as  a  redundant  GAFB 
to  Verona  link.  Since  its  path  was  direct  vs.  routed 
through  Stockbridge,  a  different  propagation  delay 
time  could  be  expected  as  compared  to  the  GAFB- 
Stockbridge-Verona  microwave  link. 

The  final  network  configuration  is  shown  in  Figure  3.1. 

The  network  consisted  of  a  three-node  tandem  configuration 
with  one  TROPO  and  one  LOS  microwave  link.  The  Verona  test 
site  was  the  middle  node.  This  is  the  only  network  in  which 
timing  experiments  were  conducted. 

3.1  Tandem  Network 


The  previous  discussion  described  how  equipment  avail¬ 
ability  reduced  the  testing  and  limited  the  network  configura¬ 
tion  possibilities  to  a  three-node,  two-link  tandem  network. 
Figure  3.2  shows  the  layout  of  this  network.  Note  that  the 
center  node  is  the  Verona  test  site.  Also,  link  conti  1  func¬ 
tions  were  carried  out  through  public  telephone  lines  as 
depicted.  In  this  subsection,  we  describe  the  equipment 
involved  with  both  the  troposcatter  and  microwave  radio  links. 
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3.2  Link  Configuration 


3.2.1  GAFB- Stockbridge -Verona  LOS  Link 

This  microwave  link  operates  at  8  GHz  using  Philco  Ford  mo¬ 
dems  (Quaternary  Baseband)  [3.2]  [ 3.3]  and  radios  (LC-8D)  Stock- 
bridge  is  an  IF  repeat  stage  only  and  no  modems  are  used.  A  dir¬ 
ect  link  between  GAFB  and  Verona  (14  GHz  TCM-608B  radio  set)  did 
not  becoae  operational  during  the  test  program  so  it  is  not  con¬ 
sidered  here  although  Figures  3.3  and  3.4  show  the  Philco- 
Ford  QPSK  microwave  modems  connected  to  the  Timing  Subsystems 
implying  that  such  a  link  was  used,  these  connections  were  made 
to  evaluate  equipment  delays  as  detailed  in  Section  4. 

Dish-to-dish  slant  range  path  length  surveys  were  per¬ 
formed  in  summer  1977  to  re-establish  site  locations  [3.1]  [3.4]. 
Although  this  information  is  not  necessary  for  frequency 
alignment  tests,  absolute  clock  (phase)  alignment  requires  an 
accurate  estimate  of  propagation  and  equipment  delays. 

Figure  3.3  depicts  the  microwave  range  path  distances. 

Note  that  the  total  path  length  is  about  25.7  miles, 

17.8  miles  from  GAFB  to  Stockbridge  and  7.9  miles  from  Stock- 
bridge  to  Verona.  The  corresponding  atmospheric  signal  delay 
is  approximately  138  fis . 

It  should  be  noted  that  although  this  is  a  two- frequency , 
full-duplex  link,  only  simplex  operation  was  available 
(Verona  to  GAFB) .  This  condition  remained  throughout  the 
testing  period  despite  efforts  by  site  personnel  to  find  the 
problem. 

The  Quaternary  Baseband  Modem  can  be  divided  into  two 
major  sections;  a  transmitter  section  and  a  receiver  section. 

The  transmitter  section  includes  all  the  circuits  necessary 
for  the  transformation  of  a  binary  signal  into  a  four-level 
baseband  waveform.  The  receiver  section  comprises  all  of  the 
circuits  necessary  to  convert  a  four  level  baseband  waveform 
into  a  decoded  and  regenerated  binary  data  signal.  A  block 
diagram  illustrating  the  interrelationship  of  these  two  major 
sections  with  each  other  and  with  the  microwave  radios  is  pre¬ 
sented  in  Figure  3.4. 
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Figure  3.3  Microwave  Tower  Survey  Points  (from  [3.1]) 
(Stockbridge,  Verona,  Bldg.  3) 
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Quaternary  Baseband  Modem  is  designed  to  interface  at 
baseband  with  a  Philco-Ford  LC-8D  or  similar  FDM  type  microwave 
radio.  The  modem  is  basically  an  AM  modulator  and  demodulator 
which  generates  and  receives  a  baseband  four-level  1  volt  p-p 
signal.  When  interfaced  with  FDM  type  microwave  radios,  the 
AM  modulator  produces  quaternary  frequency  shift  keyed  modula¬ 
tion  of  the  carrier  signal. 

All  external  connections  (except  power)  to  the  Modem  are 
made  at  the  BNC  connectors  on  the  front  of  the  module.  No 
modifications  to  the  radio  equipment  were  required  for  achieving 
satisfactory  performance. 

3.2.2  Verona-Youngstown  TROPO  Link 

This  test  range  is  comprised  of  two  test  sites  at  Youngs¬ 
town  and  Verona.  One  quadruple-diversity  AN/TRC-132A  radio  set 
is  installed  at  each  site.  This  4.4  -5.0  GHz  (C-Band)  radio 
set  can  provide  full  duplex  operation.  The  path  length  is 
approximately  168  miles  so  typical  propagation  delay  would  be 
about  910  ns. 

A  detailed  layout  of  the  AN/TRC-132  equipment  at  Youngs¬ 
town  is  shown  in  Figure  3.5.  This  diagram  shows  the  intercon¬ 
nections  between  shelters  as  well  as  each  of  the  functional 
blocks  in  the  system. 


A  Raytheon  DAR-IV  high-speed  digital  TROPO  modem  was  used 
at  each  end  of  the  link  to  provide  efficient  megabit  data 
transmission.  The  DAR  (Distortion  Adaptive  Receiver)  modem 
technique  developed  by  Raytheon  provides  several  unique  features. 
Among  these,  are  simple  QPSK  modulation,  adaptive  matched  filter 
detection,  optimum  path  diversity  combining  and  optimum  in-band 
gain.  The  modem  used  a  70  MHz  IF  to  carry  an  analog  order  wire 
signal  and  one  or  two  asynchronous  15.44  megabit  data  streams 
at  transmission  rates  of  1.75  megabits  and  3.5  megabits,  res¬ 
pectively  . see  [3.5].  The  typical  modem  diversity  configurations 
is  shown  in  Figure  3.6. 


Figure  3 . 5 


Youngstown  Troposcatter  Facilities  (TRC-132) 
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3.3  Node  Configuration 

3.3.1  Timing  Subsystem 

At  each  site,  the  Timing  Subsystems  were  connected  to  mo¬ 
dems  and  the  Frequency  Measurement  Terminal.  At  the  Youngstown 
and  Verona  sites.  It  was  necessary  to  provide  an  external  fre¬ 
quency  source  for  the  TS  because  these  units  did  not  include 
internal  quartz  oscillators.  The  rubidium  frequency  standard 
housed  in  the  FMT  was  used  for  this  purpose  as  well  as  providing 
the  frequency  source  for  the  FMT  measurement  capability.  In 
addition,  at  each  site,  the  FMT  was  configured  to  measure  and 
average  both  LORAN-C  and  TS  adjusted  output  l-pps  events. 

This  scheme  provided  the  measurement  capability  needed  to  verify 
network  performance.  Each  Timing  Subsystem  can  measure  and  aver¬ 
age  up  to  3  external  l-pps  signals,  and  as  a  redundancy  measure, 
rubidium  and  LORAN-C  l-pps  outputs  were  connected  directly  to 
the  TS  auxiliary  TRP  ports.  It  was  a  simple  matter  to  verify 
TS  tracking  performance  when  using  the  rubidium  l-pps  output  as 
a  reference  input  to  the  Timing  Subsystem.  Shakedown  tests 
demonstrated  that  both  the  FMT  and  TS  interval  measurement 
and  averaging  facilities  performed  as  expected. 

Figures  3.7,  3.8  and  3.9  depict  the  equipment  connections 
to  the  Timing  Subsystem.  Notice  that  although  the  connections 
to  the  QPSK  LOS  modems  are  shown,  the  GAFB  to  Verona  (direct) 
link  was  not  used.  However,  as  Section  4  reveals,  the  TS 
data  rates  were  wired  to  conform  to  the  radio  MUX  rates  should 
the  link  be  used  in  future  tests. 

3.3.2  Network  Controller 


The  Network  Controller  allows  central  conrrol  and  monitor¬ 
ing  of  each  Timing  Subsystem  within  a  network.  Typically, 
the  operator  would  choose  a  timing  experiment,  enter  the  para¬ 
meters  at  the  terminal  keyboard  and  download  experiments  through 
a  single  node.  Resync  and  reorganization  tasks  would  be  handled 
exclusively  by  the  TS's.  The  essence  of  this  approach  is  to 
centralize  the  setup  and  monitoring  function  to  avoid  unneces¬ 
sary  travel  to  remote  sites  merely  to  change  a  switch  setting. 
Section  2.2.2  describes  the  purpose  of  the  Network  Controller 
and  the  equipment  involved. 

During  software  development,  it  was  decided  to  abandon  the 
idea  of  committing  the  microcomputer  development  system  to  the 
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Figure  3.7  Node  Configuration  (GAFB) 
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Figure  3.8  Node  Configuration  (Verona) 


TIMING  SUBSYSTEM 


Figure  3.9  Node  Configuration  (Youngstown) 


Network  Controller  function  as  originally  intended.  This  de¬ 
cision  was  based  on  many  factors  that  became  evident  during  the 
course  of  the  program.  The  following  paragraphs  describe  some 
of  the  reasons. 

First,  shortly  after  the  design  phase,  it  was  agreed  that 
the  use  of  supervisory  and/or  nodal  management  traffic  in  addi¬ 
tion  to  that  needed  to  support  the  timing  function,  would 
unnecessarily  burden  the  channel  by  increasing  the  complexity 
to  the  point  where  actual  timing  data  would  seemingly  be  lost  in 
the  midst  of  a  nodal  management  protocol.  Granted,  a  convenient 
feature  of  the  Network  Controller  was  that  of  centralizing  net¬ 
work  control  functions,  but  the  effort  of  implementing  this 
one  feature  alone  would  result  in  significant  increases  in 
software  effort  from  an  already  large  and  comprehensive  package. 
Considering  it  was  not  a  contractually  required  function,  it  was 
agreed  to  leave  this  aspect  for  future  considerations;  i.e., 
an  advanced  development  model,  and  concentrate  on  Timing 
Subsystem  operational  software.  Additionally,  this  represented 
no  great  hardship  to  the  success  of  the  experimental  program 
because  each  TS  is  in  its  own  right  a  programmable  device  with 
its  own  simple  system  monitor  and  in  a  small  network  of 
three  modes,  a  central  monitoring  function  was  not  essential. 

In  fact,  a  solution  was  worked  out  that  was  nearly  as 
effective  and  required  almost  no  additional  equipment:  a  small, 
portable  communications  terminal  was  named  the  Network  Control¬ 
ler  and  carried  out  its  function  through  the  use  of  commercial 
telephone  lines.  This  is  indicated  in  the  diagram  of  the  tan¬ 
dem  network  as  shown  earlier.  This  was  made  easier  by  using 
the  already  existing  modems  and  data  couplers  resident  in  each 
of  the  Frequency  Measurement  Terminals  (FMT)  located  at  each 
test  site.  The  next  subsection  describes  in  more  detail  the 
role  of  the  FMT  in  the  experimental  program.  In  this  light, 

TS  interrogation  was  a  simple  matter  of  dialing  the  number  for 
the  site  FMT  and  insuring  that  the  TS  E/A  interface  was  connec¬ 
ted  to  the  FMT  as  well.  An  essentially  direct  connection  to 
the  TS  could  be  made  from  any  telephone.  Considering  the 
small  size  of  the  network  and  the  extra  mobility  and  convenience 
gained  with  a  portable  terminal,  network  control  and  test  setups 
were  never  difficult  although  occasionally,  the  telephone  lines 
themselves  Impeded  data  traffic.  One  solution  would  be  to  in¬ 
stall  a  higher  quality  or  permanent  data  line  but  it  was  un¬ 
necessary  for  this  program. 
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Figure  3.10  shows  the  network  controller  interconnection 
scheme  with  a  single  Timing  Subsystem. 

A  direct  cable  connection  to  the  Timing  Subsystem  was 
used  when  development  engineers  were  actually  located  at  a 
site.  In  effect,  the  role  of  Network  Controller  was  reduced 
to  that  of  an  interactive  monitor  because  all  control  and  debug¬ 
ging  was  performed  through  a  direct  connection  rather  than 
through  intemode  data  communication  links.  This  greatly 
simplified  some  of  the  more  mundane  tasks  of  the  experimental 
program  such  as  equipment  shakedown  and  single  node  performance 
monitoring  when  removed  from  a  network.  Performance  assessment 
during  a  network  synchronization  experiment  was  accomplished 
by  requested  memory  dumps  of  saved  averages  of  clock  performance 
stored  at  each  node.  Also,  status  requests  provided  a  quick 
method  to  check  on  the  operation  status  of  a  given  node  during 
an  experiment.  Reference  [3.6]  provides  a  complete  summary 
of  the  capabilities  of  TS  monitor  software. 

3.3.3  Frequency  Measurement  Terminal 

The  Frequency  Measurement  Terminal  (FMT)  was  developed 
under  contract  by  the  National  Bureau  of  Standards,  Time  and 
Frequency  Division.  It  is  a  flexible  frequency  measurement 
device  incorporating  an  integral  LORAN-C  receiver  and  LORAN-C 
pulse  monitoring  equipment,  a  CPU  containing  time  of  day  read¬ 
out,  time  interval  measurement  unit  as  well  as  a  digital  modem, 
serial  interface,  associated  memory,  pulse  shaping  circuits, 
printing  terminal,  and  finally  a  rubidium  frequency  standard. 

All  equipment  is  housed  in  a  single  width  rack. 

Besides  being  capable  of  monitoring  LORAN-C  or  TV  Line  10 
performance  as  compared  with  the  local  rubidium  standard,  the 
FMT  was  used  as  an  integral  part  of  the  general  node  performance 
assessment  scheme.  The  FMT  has  the  ability  to  measure  lpps ‘ 
pulses  on  as  many  as  5  input  channels  compared  to  the  source 
lpps  pulse  provided  by  the  rubidium  standard.  Therefore,  not 
only  may  the  local  rubidium  frequency  standard  be  used  as  a 
node  reference  but  by  connecting  the  adjustable  lpps  output 
from  the  Timing  Subsystem  to  the  FMT,  and  in  turn,  connecting 
the  frequency  standard  output  lpps  directly  to  the  TS  (while 
leaving  the  FMT  source  lpps  from  the  frequency  standard  Intact) 
for  time  interval  measurement,  a  redundant  TS  to  rubidium 
measurement  facility  is  established.  This  also  provides  an 
easy  way  to  compare  the  TS  adjustable  output  to  any  other 
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Figure  3.10  Network  Controller  to  Timing  Subsystem  Interconnection  Scheme 


externally  applied  1-pps  signal  (such  as  LORAN-C) .  Once  the 
relative  drift  between  say,  a  received  LORAN-C  pulse  and  the 
local  rubidium  standard  is  known,  then  the  Timing  Subsystem 
adjustable  clock  may  be  compared  to  the  LORAN-C  pulse  for  per¬ 
formance  evaluation.  This  is  precisely  the  technique  used 
during  network  synchronization  test. 

The  FMT  uses  an  internal  time  Interval  counter  with  a 
100-MHz  time  base  to  yield  a  resolution  of  10  ns  in  the  least 
significant  digit.  Together  with  a  plus  or  minus  one-count 
ambiguity  inherent  to  the  counter  Itself,  other  internal  factors 
contribute  to  an  uncertainty  of  about  SO  ns  in  each  individual 
time  interval  measurement;  but  some  external  factors  must  be 
considered  also. 

The  accuracy  of  FMT  calibrations  is  most  certainly  limited 
by  the  instability  of  TV  and  LORAN-C  propagation  paths.  If 
two  terminals  are  within  line-of-sight  range  of  the  same  TV 
station,  their  differential  path  delay  might  vary  by  as  much 
as  20  ns.  If  the  terminals  are  separated  by  a  greater  distance 
so  that  a  TV  microwave  distribution  prth  is  involved,  the 
differential  delay  may  occasionally  change  by  several  hundred 
microseconds.  By  comparing  the  results  from  three  dif¬ 
ferent  networks,  however,  the  uncertainty  in  TV  propagation 
delay  can  be  resolved  to  within  100  ns  generally. 

LORAN-C  is  capable  of  supporting  absolute  timing  accuracy 
of  about  100  ns  over  long  distances.  If  differential  measure¬ 
ments  are  made  at  sites  separated  by  only  400  kilometers  or 
less,  the  results  using  LORAN-C  are  apt  to  be  substantially 
better. 

It  is  expected  that,  by  proper  choice  of  statistical 
weighting  factors  for  LORAN-C  and  TV  data,  the  overall  uncer¬ 
tainty  in  calibrations  using  the  FMT  will  fall  between  10  ns 
and  100  ns.  A  more  reliable  estimate  of  uncertainty  will  depend 
ultimately  upon  further  operating  experience  with  the  system 
and  a  thorough  analysis  of  actual  measurement  data.  See  [3.7]. 

3.4  Test  Bed  Master  Clock 

Each  Frequency  Measurement  Terminal  (one  located  at  each 
of  three  nodes)  receives  LORAN-C  radio  signals  from  a  single 
source:  the  Northeast  chain  master  at  Seneca,  NY.  This  station 
with  a  peak  pulse  power  of  800  kW  is  located  within  120  miles 


of  each  of  Che  receivers  at  the  test  sites.  Since  the  LORAN-C 
receivers  in  the  FHT's  were  receiving  Seneca,  its  timing  pulses 
would  be  a  convenient,  accurate,  and  traceable  network  reference 
to  which  TS  performance  may  be  compared. 

The  high  quality  of  LORAN-C  frequency  and  time  signals  is 
due  to  a  number  of  things.  Each  station  is  controlled  by  a 
cesium  standard.  Careful  control  and  monitoring  of  the  signals 
by  the  U.S.  Naval  Observatory  allows  a  user  to  achieve  rapid, 
accurate  results.  Groundwave  signals  provide  state-of-the-art 
frequency  and  time  calibrations.  As  is  the  case  with  the 
Test  Bed  network,  groundwave  signals  may  be  reliably  received 
at  distances  of  1500  miles  or  more  for  a  station  of  at  least 
300  kw  peak  pulse  power.  For  the  test  program,  we  were  con¬ 
cerned  only  with  groundwave  performance. 

The  ability  of  the  LORAN-C  timing  receiver  to  indicate 
accurate  time  readings  is  dependent  upon  the  relative  strength 
of  the  received  signals  and  the  level  of  the  local  noise  inter¬ 
ference.  As  a  result,  accuracy  deteriorates  at  receiver  loca¬ 
tions  far  away  from  the  stations.  For  timing  purposes,  long¬ 
term  averaging  is  used  to  achieve  more  reliable  results.  This 
is  precisely  the  function  of  the  FMT.  Up  to  eight  days  of  4- 
hour  averages  of  pulse  receive  time  vs.  local  time  reference 
(on-site  rubidium)  at  one  sample  per  second,  are  saved  in  the 
processor.  Armed  with  this  information,  one  may  easily  deter¬ 
mine  the  relative  frequency  drift  between  each  of  the  local 
timing  references  and  the  Seneca  LORAN-C  timing.  This  is 
the  basis  for  demonstrating  network  performance.  Therefore, 
the  network  frequency  standard  may  be  considered  to  be  the 
cesium  beam  atomic  frequency  standard  located  at  Seneca,  NY. 

If  desired,  frequency  drift  comparisons  may  be  made  directly 
to  NBS  standards,  in  Boulder,  Colorado. 

LORAN-C  is  one  of  the  principal  methods  of  intercomparing 
the  atomic  standards  at  NBS,  the  USNO  and  the  National  Research 
Council  in  Canada.  This  scheme  is  also  extended  to  the  European 
LORAN  network  and  affords  a  means  for  the  BIH  to  compare  all 
of  the  world's  main  time  scales.  The  results  reported  for 
this  system  of  intercomparison  are  impressive  -  approaching 
1  part  in  10^  over  several  months  [3.8] . 

Typical  values  for  groundwave  reception  over  land  could 
easily  approach  a  few  parts  in  10^-2  for  a  one-day  average. 
Corrections  to  LORAN  data  are  provided  by  the  USNO,  and  the  NBS 
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also  monitors  and  reports  daily  LORAN  phase  readings.  Users 
who  want  very  accurate  results  from  LORAN-C  must  correct  the  ob¬ 
served  data  by  using  the  published  corrections.  This  informa¬ 
tion  is  available  by  TWX  on  a  daily  basis  from  the  USNO. 

For  timing,  the  accuracy  of  the  groundwave  at  a  particular 
location  depends  upon  several  factors  that  contribute  to  the 
errors  in  the  measurements.  The  approximate  errors  are: 

1.  Error  in  transmitted  signal,  <  0.2  microsecond. 

2.  Errors  caused  by  propagation  path,  up  to  0.2  micro¬ 
second  over  water;  up  to  1  microsecond  over  land. 

3.  Receiver  error,  0.02  microsecond  typical. 

4.  Errors  caused  by  atmospheric  noise  and  inter¬ 
ference,  up  to  0.5  microsecond  for  interference. 

Longer  averaging  times  can  be  used  with  noisy 
signals . 
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SECTION  4 


EQUIPMENT  DELAY  MEASUREMENTS 


This  section  presents  the  findings  of  equipment  delay 
measurements  performed  as  part  of  the  equipment  shakedown  tests 
at  each  site  prior  to  network  timing  experiments.  Equipment 
delay  times,  as  well  as  path  propagation  times,  must  be  known 
or  measured  in  advance  for  any  network  timing  experiment  re¬ 
quiring  absolute  clock  alignment  (i.e.,  zero  phase  offset). 
Although  not  essential  to  network  performance  where  frequency 
tracking  is  the  desired  result,  these  measurements  further  re¬ 
inforce  methods  of  verfiying  system  operation. 

A  word  of  caution  is  in  order;  path  and  equipment  delay 
reciprocity  is  a  function  of  both  the  propagation  medium  and 
link  transmission  equipment.  Although  recent  field  experiments 
have  shown  that  path  delay  (on  the  average)  may  be  assumed  equal 
in  both  directions  [4.1],  it  is  wise  to  measure,  or  at  least 
know,  the  delay  differential  arising  from  transmission  direc¬ 
tions  as  well  as  the  various  equipments  and  frequencies  used. 

4.1  Equipment  Delay  Measurement  Procedures 

Briefly  described  is  the  method  used  to  obtain  precise 
delay  information  of  both  the  Timing  Subsystem  itself  as  well 
as  any  external  equipment,  such  as  modems  and  radios. 

The  Timing  Subsystem  has  the  capability  to  measure  the  time 
difference  between  outgoing  and  incoming  Time  Reference  Pulses 
(TRP's)  that  leave  or  enter  the  TS  at  the  link  termination 
points.  The  node  control  computer  extracts  timing  information 
from  bit  patterns  on  the  serial  bit  streams  to  and  from  the  TS. 

An  error  computation  is  performed,  and  the  TS  determines  the 
elapsed  time  from  a  transmitted  TRP  to  a  corresponding  received 
TRP.  Through  this  mechanism,  TS  as  well  as  external  equipment 
delays  may  be  determined.  Measurement  resolution  is  +10  ns 
as  dictated  by  the  100-MHz  counters  on  the  time  interval  measure¬ 
ment  board. 
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4.2  Timing  Subsystem  Equipment  Dela^ 


This  subsection  describes  steps  necessary  to  measure  equip¬ 
ment  delay  times  of  the  Timing  Subsystem  and  any  interconnected 
modems  or  radios  in  the  data  path.  For  simplicity,  only  con¬ 
nections  to  link  1  are  detailed.  However,  be  reminded  that  for 
each  link  used,  all  lower  numbered  links  must  either  be  in  use 
or  connected  back-to-back;  that  is,  if  link  2  only  is  to  be  used, 
link  1  must  be  strapped  back-to-back  for  proper  operation  of 
link  2.  This  is  necessary  because  each  link's  transmit  interrupt 
sequence  starts  the  next  link's  sequence.  (See  Section  3.2.3 
of  [4.2]  for  more  information  on  link  transmit  service.) 

Connecting  a  communications  path  to  the  Timing  Subsystem  is 
a  straightforward  matter.  The  node  synthesizers  are  wired  to 
provide  the  data  rates  described  in  Table  4-1.  These  are  the 
data  rates  used  in  the  tandem  network  configured  (also  shown  in 
this  table)  during  the  field  test  portion  of  this  program. 

To  find  the  equipment  delay  through  a  link,  refer  to  Fig¬ 
ure  4.1  for  the  recommended  equipment  interconnection  scheme. 
Figure  4.2  details  link  connections  (back  panel)  required  to 
measure  equipment  delay  of  the  TS  itself.  By  inspecting  the 
nodal  error  terms,  equipment  delay  may  be  found  as  a  32-bit 
fixed-point  (hexadecimal)  number  where  one  count  equals  10  ns 
(i.e.,  normalized  to  10“°  seconds).  See  [4.2]  and  the  clock 
data  TRIP  format  shown  in  Section  2  to  locate  the  raw  bit  pat¬ 
terns  used  to  interval  computation.  The  Timing  Subsystem  auto¬ 
matically  saves  a  series  of  elapsed  time  delay  measurements  for 
inspection  at  the  operator's  convenience.  Again,  Section  4.3  of 
[4.2]  describes  the  detailed  data-logging  procedures.  They  are 
omitted  here  but  a  summary  of  TS  back-to-back  equipment  delays 
is  in  Table  4-2. 

Note,  for  instance,  that  each  of  the  measured  back-to-back 
Timing  Subsystem  delays  seems  to  hover  around  the  500-jiS  mark. 
This  corresponds  well  to  an  expected  value  since  the  4-kHz 
(250-ps  period)  data  rate  determines  the  bit-detection  times. 

That  is,  the  round -'.rip  delays  as  shown  in  the  table  are  made  up 
of  four  components:  one  bit  period  transmit  time,  one  bit 
period  receive  time,  mux  time,  and  demux  time  for  a  typical  value 
(LOS)  of  250  ps,  250  ns,  0.5  ^s ,  and  0.5  ns >  respectively.  Other 
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Figure  5.3  Self -Reference  Test  Setup 


Mission  Bit  Stream  (MBS)  rates  show  slight  changes  in  delay,  as 
seen  in  Table  4-2.  All  4-kHz  data  (at  TS)  rates  are  determined 
by  the  node  synthesizer  which  is  phase- locked  to  the  node  stan¬ 
dard  5 -MHz  frequency  source. 

4.3  Line-of-Sight  Equipment  Delay 

The  single  LOS  route  included  two  Philco-Ford  baseband 
modems  and  four  Philco-Ford  LC-8D  microwave  radios.  An  IF 
repeat  station  was  configured  at  Stockbridge.  Total  path  length 
was  about  25.7  miles. 

In  order  to  measure  equipment  delays  throughout  the  link, 
each  portion  was  isolated.  Figure  4.3  shows  the  LOS  back-to-back 
setup  including  modems  and  radios.  Actually,  three  interconnec¬ 
tion  schemes  were  used  to  determine  individual  equipment  delays 
at  each  site.  First,  the  Timing  Subsystem  delay  was  measured. 
Second,  the  modem  was  corrected,  and  TS + modem  delay  was  measured. 
Third,  the  radio  set  was  added  and  TS + modem + radio  delay  was 
measured.  Finally,  modem  and  radio  (including  associated  cabling) 
delays  were  extracted  by  subtraction.  The  dotted  lines  in 
Figure  4.3  show  the  stages  of  interconnection  used  to  isolate 
equipment  delay.  Remember,  the  sum  total  of  TS + modem + radio 
delay  includes  both  transmit  and  receive  sections;  any  difference 
in  delay  will  result  in  a  differential  that  would  manifest  itself 
as  a  phase  offset  when  performing  double-ended  measurements.  For 
most  single-ended  schemes,  frequency  averaging  is  the  desired 
result;  therefore,  equipment  delay  compensations  need  not  be 
inc luded . 

4.3.1  LOS  Modem  Delay 

Back-to-back  delay  tests  were  carried  out  for  the  Philco- 
Ford  Quaternary  Baseband  Modems  [4.3]  when  connected  to  the 
Timing  Subsystem. 

First,  it  should  be  recognized  that  the  Timing  Subsystem 
requires  a  transmit  clock  from  the  modem  for  its  own  transmit 
mux.  However,  it  was  found  that  the  modem  lost  receiver  sync 
whenever  the  Timing  Subsystem  was  not  transmitting  data;  that  is, 
no  provision  was  made  to  provide  bit  transitions  between  TRIP 
bursts.  It  was  decided  to  incorporate  a  PN  generator  at  each 
transmit  mux  in  the  TS  to  keep  the  modem  receiver  in- lock.  Sub¬ 
sequently,  this  simple  modification  was  wired  in  and  the  problem 
was  eliminated. 
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Figure  4.3  LOS  Back-to-Back  Test  Setup  for  Delay  Measurement 


Secondly,  there  are  clock* phasing  controls  on  the  front  of 
the  modem,  one  for  the  transmit  section  and  one  for  the  receive 
section.  It  is  known  that  a  timing  variation  of  tens  of  nano¬ 
seconds  can  be  introduced  by  these  controls  within  the  range  of 
satisfactory  modem  performance  and,  overall,  a  whole  bit  of  vari¬ 
ation  is  possible.  Once  the  equipment  delay  has  been  measured, 
the  knob  setting  must  remain  intact  for  all  subsequent  timing 
technique  tests. 

The  measurements  for  both  the  GAFB  and  Verona  equipment  are 
shown  in  Table  4-3.  The  Timing  Subsystem  delay  component  was 
measured  separately  and  subtracted  from  back-to-back  tests  with 
the  modem  transmitter  connected  directly  to  the  receiver.  The 
resulting  delay  values  are  shown  in  Table  4-3. 

No  delay  uncertainty  was  anticipated  for  this  equipment 
because  it  does  not  include  an  internal  multiplexer.  Measure¬ 
ment  resolution  was  10  ns . 

4.3.2  LOS  Radio  Delay 

Delay  measurements  were  carried  out  on  the  LOS  radios 
(Philco-Ford  LC-8D)  at  the  Griffiss  and  Verona  sites.  Although 
Stockbridge  was  used  as  an  IF  repeat  station  for  the  microwave 
link,  a  measurement  capability  was  not  available  there;  there¬ 
fore,  delay  figures  based  on  measurements  from  the  other  radios 
and  [4.1]  were  used. 

The  equipment  configuration  for  the  GAFB  radio  is  identical 
to  that  shown  in  Figure  4.3  described  earlier.  Extracting  radio 
delay  required  connecting  both  the  TS  and  modem,  whose  delays 
were  previously  determined,  to  the  radio.  Then,  the  measured 
delay  of  all  of  the  equipment,  less  that  of  the  TS  and  modem, 
left  pure  radio  delay. 

Since  the  measurements  for  this  program  agreed  quite  well 
with  those  of  a  previous  CNR  effort,  presented  here,  are  the  data 
for  the  GAFB,  Verona,  and  Stockbridge  sites  in  Table  4-4.  In 
that  program,  the  test  setup  for  the  Verona  and  Stockbridge 
radios  involved  injecting  baseband  tones  into  the  modulator. 

The  tones  ranged  from  50  kHz  to  1500  kHz.  The  loopback  was  at 
the  RF  output  level.  Both  A  and  B  side  delays  are  included. 

For  the  GAFB  radio,  the  measurement  resolution  was  10  ns,  as 
dictated  by  the  TS. 
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TABLE  4-3 


DELAY  MEASUREMENTS  FOR  PHILCO-FORD  BASEBAND  MODEM 


Modem  #1  (GAFB)  Modem  #2  (Verona) 

TS  #001  TS  #003 


Data 

Rate 

3.088  Mb/s 

3.088  Mb/s 

Scrambler 

Off 

Off 

Receiver 

Timing 

Adjust 

Fully 

CCW 

Fully 

CCW 

Transmit 

Timing 

Adjust 

Fully 

CCW 

Fully 

CCW 

TS  + Modem 

505.02  ns 

505.06  ps 

Modem 

4.03  ms 

* 

4.05  ms 

Any  cabling  between  the  modem  and  TS  was  included 
as  part  of  the  modem  delay. 
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4.4  Troposcatter  Equipment  Dela1 


The  TROPO  route  from  Youngstown  to  Verona  was  configured  as 
a  full-duplex  C-band  link  using  DAR-IV  modems  and  the  AN/TRC-132A 
radio  set.  The  equipment  configuration  at  each  site  was 
identical. 

First,  the  delay  through  the  modem  in  loopback  mode  was  in¬ 
vestigated.  Internal  modulator /demodulator  data  streams  were 
intercepted  to  provide  framing  and  signaling  formats  required 
by  the  Timing  Subsystem.  Delay  was  measured  by  looping  back  at 
IF.  Second,  the  radio  set  delay  was  investigated.  In  the  effort 
of  saving  time  and  concentrating  on  the  experimental  program, 
radio  delay  was  not  actually  measured;  if  required,  the  results 
from  a  similar  measurement  series,  performed  on  the  same  equip¬ 
ment  [4.1],  were  used. 

4.4.1  TROPO  Modem  Delay 

The  DAR  (Distortion  Adaptive  Receiver)  modem  is  designed  to 
provide  efficient  megabit  data  transmission  over  troposcatter 
links.  It  was  used  in  quad  diversity  mode  at  a  3.5-Mb/s  trans¬ 
mission  rate  (dual  mode) .  Minor  modifications  were  made  to  this 
equipment  to  extract  timing  and  framing  information  in  addition 
to  basic  signal  level  conversion  to  meet  TS  link  termination 
requirements.  Modifications  allowed  the  Timing  Subsystem  to 
clock  the  DAR  multiplexer.  The  multiplexer  combines  two  1.544- 
Mb/s  user  data  streams  with  a  192-kb/s  orderwire  channel  to  form 
a  3.5-Mb/s  data  output  stream.  To  eliminate  the  timing  ambi¬ 
guities  in  passing  through  the  multiplexer,  the  TS  output  data 
stream  was  applied  directly  to  the  QPSK  modulator.  Also,  the 
mux  output  clock  from  the  DAR  was  used  to  correctly  time  data 
into  the  QPSK  modulator. 

In  addition,  the  Timing  Subsystem  frequency  synthesizer  was 
wired  to  generate  a  10-MHz  clock  for  the  modem  so  that  proper 
data  phasing  would  be  possible.  Finally,  since  the  multiplexer 
was  bypassed  in  the  modem  transmitter,  the  demultiplexer  was 
bypassed  in  the  modem  receiver.  See  [4.4]  for  a  description  of 
the  DAR  modem. 

Occasional  receiver  sync  problems  were  handled  by  sending  a 
pulse  to  the  modem  to  force  receiver  resync.  Due  to  the  long 
tracking  loop  time  constants,  this  action  was  needed  only 
periodically.  A  receiver  loss-of-sync  light  on  the  modem 
indicated  the  need  to  resync. 


Figure  4.4  shows  the  TS  modem  back-to-back  setup  to  measure 
modem  delay  (TS  delay  was  measured  first) .  Table  4-5  shows  the 
results  of  this  measurement  series. 

4.4.2  Delay  Characteristics  of  the  TRC-132A  Radios 

In  an  effort  to  obtain  a  better  understanding  of  the  delay 
variability  occurring  in  the  receiver  and  transmitter  components 
of  the  system,  a  comprehensive  set  of  tests  was  undertaken  by 
CNR  in  a  previous  effort  [4.1].  The  use  of  a  variable  frequency 
tone  probing  technique  was  dictated  by  the  desire  to  obtain 
preliminary  equipment  delay  data  prior  to  the  availability  of 
the  digital  pseudo-noise  (PN)  phase-encoded  probing  system. 
Furthermore,  this  approach  allowed  measurement  of  the  FM  modu¬ 
lator  and  demodulator  portions  of  the  radio  which  had  a  low 
frequency  cutoff  at  around  100  kHz,  thereby  ruling  out  interfaces 
with  the  PN  equipment. 

The  main  objective  was  to  isolate  the  various  delay  com¬ 
ponents  in  the  transmitter  and  receiver  as  far  as  possible,  and 
to  establish  the  delay  sensitivity  to  carrier  offset,  tempera¬ 
ture,  and  probing  tone  frequency.  Moreover,  it  was  expected  that 
differences  in  radios  of  the  same  family  would  also  be  observed. 

Some  of  the  delay  measurement  data  are  presented  in 
Table  4-6,  where  the  back-to-back  radio  delay  is  shown  as  a 
function  of  unit,  carrier  frequency,  and  tone  frequency.  All 
of  these  measurements  were  taken  with  the  narrowband  TRC-132A 
filters  in  place. 

4.5  Tandem  Network  Delay  Elements 

To  summarize,  the  findings  of  the  equipment  delay  measure¬ 
ment  program  are  presented  in  roadmap  form.  Note  that  single 
direction  delays  are  present;  that  is,  half  the  magnitude  of  the 
results  of  the  back-to-back  (transmit  -  receive  tests.  From  the 
findings  of  the  CNR  effort  entitled  "System  Timing  and  Synchron¬ 
ization"  [4.1],  we  may  assume  that  path  delays  are  essentially 
the  same  in  both  directions  (on  the  average).  This  is  a  critical 
assumption  and  one  that  may  be  credited  for  the  success  of 
supporting  a  timing  function  in  full-duplex  operation.  As 
stated  earlier,  differential  delay  differences  for  equipment 
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j  FREQUENCY 
i  STANDARD 


MODIFIED  TO  PROVIDE  FRAMING  AND 
SIGNAL  LEVEL  CONVERSION 


Figure  4.4  TROPO  Modem  Back-to-Back  Setup 
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TABLE  4-5 


DELAY  MEASUREMENTS  FOR  DAR-IV  TROPO  MODEM 


Modem  #6  (Verona)  Modem  #4  (Youngstown) 

TS  #003  TS  #002 


Data 

Rate 

1 

3.5  Mb/s  j 

3  5  Mb/s 

' 

Xmit 

Osc  In 

10  MHz 

10  MHz 

Mode 

Select 

Dual 

Dual 

OG  Alarm 

Normal 

Normal 

TS  4-  Modem 
+  Cables 

507.20  ns 

507.10  ms 

Modem 

6.19  ns 

* 

6.09  ms 

Any  cabling  between  the  modem  an;*  TS  was 
included  as  part  of  the  modem  delay. 


TABLE  4-6 


From  [4.2] 


will  be  manifested  as  phase  error  when  processing  double- 
ended  transfers.  Since  it  was  not  possible  to  measure  equip¬ 
ment  delay  other  than  in  back-to-back  modes,  unidirectional 
delay  could  not  be  extracted  experimentally.  For  most  purposes, 
assuming  equal  delay  in  either  direction  is  adequate  and  may  be 
found  by  halving  the  round-trip  delay.  Figure  4.5  depicts 
the  equipment  and  path  delay  elements  for  signal  travel  in 
either  direction. 
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SECTION  5 


PERFORMANCE  TESTS  AND  DATA  PRESENTATION 


In  this  section  a  review  of  each  of  the  candidate  timing 
techniques  as  applied  to  network  synchronization  as  well  as 
laboratory  and  field  tests  performed  using  portions,  or  all, 
of  the  tandem  network  described  in  Section  3  is  presented. 

Section  5  closes  with  an  interpretation  of  obtained  results 
including  methods  used  to  verify  clock  performance. 

5.1  Time/Frequency  Synchronization  Hierarchy 

Many  of  the  system  and  link  control  techniques  have  common 
features.  The  intention  of  this  section  is  to  summarize  the 
important  parameters  and  issues  for  a  representative  number  of 
the  suggested  candidates.  These  candidates  may  be  broadly 
classified  into  Relative  Time  Synchronization  and  Precise  Time 
Dissemination  types. 

It  will  be  assumed  that  the  reader  is  reasonably  conversant 
with  each  method,  thereby  allowing  the  presentation  to  be  more 
concise.  References  to  the  many  relevant  synchronization  studies 
that  have  been  funded  in  the  past  are  provided  for  the  reader  who 
needs  more  background.  For  a  direct  comparison  of  the  alterna¬ 
tive  network  timing  and  synchronization  techniques,  the  reader  is 
referred  to  [5.1].  Despite  the  abundance  of  information  avail¬ 
able  on  the  various  network  synchronization  techniques,  a  formal 
classification  scheme  does  not  seem  to  have  evolved. 

The  classification  scheme  outlined  here  serves  the  purpose 
of  defining  the  various  synchronization  conditions  under  which 
a  system  may  operate. 

In  Table  5-1,  we  have  classified  five  levels  of  network 
synchronization,  beginning  with  the  lowest  level  of  independent 
uncoordinated  clocks  and  progressing  through  to  the  highest  level 
which  involves  universal  time  dissemination.  There  are  two  main 
groups:  one  collection  lumped  under  "relative  time  synchroniza¬ 
tion",  and  a  second  called  "precise  time  dissemination".  The 
former  group  represents  techniques  which  are  sufficient  to 
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TABLE  5-1 


Level  1: 


Level  2: 


Level  3: 


Level  4; 


Level  5: 


TIME /FREQUENCY  SYNCHRONIZATION  HIERARCHY 


Independent  Node  Clocks 
No  coordination 

Frequency  Synchronization 

All  clocks  at  same  average 
frequency;  system  insensi¬ 
tive  to  phase  steps 

Relative  Time  Synchronization 

Coordination  of  time  reference 
at  each  node  with  incoming 
time  reference  events 


) 


Relative  Time 
Synchronization 


Network  Time  Reference  Sync  1 


All  clocks  in  network  synchro¬ 
nized  to  internal  network  time 


Precise  Time 
Dissemination 


Time  Synchronization  with  an 
External  Reference  (e.g.. 
Universal  Time) 
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satisfy  network  synchronization  for  communications  purposes, 
while  the  latter  techniques  offer  the  enhancement  of  a  precise 
time  reference  at  every  node.  Further  subdivisions  are  possible 
depending  on  the  detailed  node  and  link  distribution  scheme. 

Figure  5.1  further  refines  the  generic  time  and  frequency 
synchronization  technique  categorization.  The  distinction 
between  precise  time  and  relative  time  is  in  the  sense  pre¬ 
viously  discussed;  the  relative  time  signals  are  lacking  in 
known  ambiguity  to  the  resolution  desired,  e.g.,  years,  days, 
seconds,  or  cycles.  Independent  clocks,  pulse  stuffing,  and 
mutual  sync  all  qualify  as  relative  time  synchronization 
techniques  which  involve  a  fixed  structure.  Adaptive  relative 
time  distribution  and  the  so-called  relative  time  distribution 
tree  are  two  variations  of  the  Master/Slave  (M/S)  concept. 

For  the  latter  category,  an  invariant  time  tree  is  selected 
for  the  network  so  that  all  clocks  are  related  to  a  single 
master,  either  by  direct  Master/Slave  connection  or,  after  a 
succession  of  Master/Slave  links,  through  the  network.  The  adap¬ 
tive  scheme  differs  primarily  in  that  it  is  possible  to  dynami¬ 
cally  alter  the  Master/Slave  tree  configuration.  Usually,  such  a 
scheme  is  categorized  as  a  self-organizing  Master/Slave  type,  and 
the  best  example  of  an  actual  implementation  is  the  Canadian 
DATAROUTE.  A  similar  breakdown  is  possible  for  the  precise  time 
synchronisation  candidates.  Basically,  there  are  fewer  choices 
here  because  the  idea  of  time  transfer  implicitly  involves  the 
use  of  a  master  reference;  one  definition  of  time  must  be  agreed 
upon,  and  all  nodes  then  attempt  to  slave  to  this  reference. 

The  adaptive  precise  time  reference  distribution  approach 
bears  the  same  relationship  to  the  precise  time  distribution  tree 
as  discussed  for  the  analogous  relative  time  techniques,  i.e., 
the  former  is  able  to  reorganize  in  times  of  adversity  so  that 
while  the  master/slave  structure  is  retained,  the  choice  of 
master  is  not  fixed,  and  the  distribution  of  timing  is  over  a 
variable  tree  structure. 

Finally,  it  should  be  observed  that,  in  Figure  5.1,  ano¬ 
ther  level  of  system  distinction  for  five  of  the  listed  net¬ 
work  synchronization  schemes  is  shown.  Refer  in  Figure  5.1  to 
the  choice  of  double-  or  single-ended  link  synchronization.  In 
summary,  one  can  say  that  this  distinction  is  between  techniques 
involving  an  exchange  of  measurements  between  nodes,  thereby 
eliminating  most  of  the  clock,  phase,  or  time  shift  attributable 
to  path  length,  and  the  alternative  of  calibrating  and  averaging 
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out  path  effects .  A  good  example  of  this  is  the  transfer  of 
time  or  frequency  via  a  (stationary)  satellite  path.  If  the 
satellite  ephemeris  is  known,  the  delay  of  the  received  signal 
relative  to  the  system  reference  clock  can  be  computed  and  sub¬ 
tracted  out  to  form  a  clock  error.  With  two-way  transmissions 
and  exchange  of  clock  error  signals  via  a  data  link,  path  length 
changes  can  be  accounted  for  without  such  detailed  calculations. 
See  [5.1]  -  [5.4]  for  information  on  this  subject. 

5.2  Candidate  Timing  Techniques 


The  main  thrust  of  this  program  was  to  design,  fabricate, 
and  test  three  experimental  models  of  candidate  Timing  Sub¬ 
systems.  Each  prototype  provided  the  capability  of  operating 
in  one  of  the  five  system  timing  modes  of  operation.  These  five 
modes  are: 

•  Independent  Clock 

•  Mas ter/S lave 

•  Mutual  Synchronization 

•  Time  Reference  Distribution 

•  Improved  Time  Reference  Distribution 

This  section  describes  each  of  the  timing  technique  candi¬ 
dates  as  implemented  in  the  Timing  Subsystem  processor.  The 
reader  is  referred  to  [5.4]  for  a  treatment  of  the  software 
involved  in  each  of  these  techniques.  A  presentation  of  the 
Time  Reference  Distribution  concepts,  as  interpreted  and  imple¬ 
mented  by  CNR,  may  be  found  in  Appendix  A.  In  this  section 
are  details  of  some  of  the  more  important  attributes  of  Time 
Reference  Distribution  as  compared  to  the  other  candidates. 

5.2.1  Independent  Clocks 

This  constitutes  the  simplest  of  all  techniques  for  synchro¬ 
nization,  at  least  from  a  conceptual  point  of  view.  The  indi¬ 
vidual  nodal  clocks  are  assumed  to  be  sufficiently  stable  so  that 
link  buffers  of  reasonable  length  can  maintain  bit  integrity  over 
a  period  of,  say,  50  days.  Clearly,  the  buffer  is  filled  or 
depleted  at  a  rate  determined  by  the  difference  in  nodal  flock 
and  incoming  data  rates. 

The  accumulation  of  data  continues  until  the  buffer  over¬ 
flows  or  underflows,  and  the  time  to  reach  this  condition  is 
determined  by  the  clock  stability  and  path  delay  variation,  as 
well  as  by  the  length  of  the  buffer.  Even  clocks  with  the  same 


long-term  average  frequency  will  show  an  apparent  Instantaneous 
difference  at  the  buffer  as  a  result  of  propagation  delay  varia¬ 
tions,  and  when  there  Is  a  constant  average  frequency  offset 
between  the  two  node  clocks,  the  elastic  store  fills  or  depletes 
linearly  with  time.  The  operation  of  resetting  the  buffer  when 
It  overflows  naturally  gives  rise  to  detrimental  system  effects; 
data  Is  lost  (or  repeated)  resulting  In  loss  of  Bit  Count  Integ¬ 
rity  (BCI)  and  synchronization  difficulties.  Hence,  the  level  at 
which  the  buffer  is  introduced  is  a  critical  choice  given  the  BCI 
effects  that  high-level  demux's  have  on  lower-level  units  at  the 
time  of  sync  loss. 

The  best  approach  that  can  be  adopted  for  totally  indepen¬ 
dent  node  clocks  is  to  use  highly  stable  references  such  as 
atomic  clocks,  with  a  large  enough  buffer  to  maintain  BCI  over 
a  predetermined  period.  After  loss  of  BCI,  the  system  must  be 
reinitialized. 

There  are  several  types  of  frequency  standards  in  common 
use  today,  including  the  cesium  atomic  beam  resonator,  the  rubi¬ 
dium  gas  cell  resonator,  and  the  quartz  crystal  oscillator.  The 
first  of  these  is  a  primary  standard  giving  good  long-term  sta¬ 
bility  without  requiring  any  other  reference  for  calibration. 

The  last  two  are  designated  secondary  standards,  meaning  that 
calibration  of  some  kind  is  necessary  at  intervals,  depending 
on  the  desired  accuracy.  The  two  mentioned  above  have  the  ad¬ 
vantage  of  compactness  and  portability  in  contrast  to  cesium 
standard.  The  three  types  of  frequency  standards  are  compared 
in  Table  2-1  and  discussed  in  Section  2. 

5.2.2  Mas ter/S lave 

Networks  employing  a  fixed  Master/Slave  timing  distribution 
contain  a  single  predetermined  master  clock  chosen  at  the  outset 
of  a  timing  experiment.  The  Master/Slave  technique  used  during 
this  program  is  the  fixed  hierarchy  type  and  is  not  as  demanding 
in  terms  of  clock  stability  as  the  independent  clock  method. 

The  key  idea  is  that  individual  nodes  take  their  timing 
information  from  only  one  of  the  terminated  links  at  that  node 
in  a  prearranged  way.  The  choice  is  made  so  that  timing  signals 
originating  at  the  master  clock,  are  disseminated  through  the 
network  via  interconnected  links,  each  slave  deriving  its  clock 
control  from  only  one  of  those  links,  and  passing  timing  infor¬ 
mation  to  other  directly-connected  nodes.  This  is  handled  in 
such  a  way  that  every  node  receives  a  suitable  reference  from  a 
single  link  connected  to  one  of  its  neighbors. 
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Master/Slave  synchronization  is  defined  as  a  system  using 
directed  control  in  which  timing  information  is  distributed  in 
only  one  direction  over  any  duplex  link.  The  direction  of  con¬ 
trol  is  always  away  from  the  designated  master,  and  transmission 
time  delays  or  equivalent  delays  not  accounted  for  will  result 
in  an  absolute  time  offset  relative  to  the  master.  However,  M/S 
is  a  frequency-averaging  technique  and  the  long-term  average 
frequency  of  all  nodes  should  be  the  same. 

For  the  field  test  platform,  network  masters,  chosen  for 
the  M/S  experiments,  were  either  an  internal  oscillator  in  the 
TS  or  an  external  reference  such  as  LORAN-C  or  laboratory  fre¬ 
quency  standard.  Using  such  a  technique  may  provide  time  dis¬ 
semination  where  the  accuracy  is  dependent  on  system  time 
transfer  parameters  and  more  general  delay  elements.  Timing 
Subsystem  hardware  was  fully  capable  of  supporting  either  scheme 
for  network  master.  Generally,  in  a  Master/Slave  network  timing 
experiment,  an  external  (but  local)  frequency  source  was  con¬ 
nected  to  the  TS,  and  the  TS  was  programmed  to  track  the  1-pps 
output  of  the  frequency  source.  This  TS  was  designated  net¬ 
work  master. 

Setting  up  an  experiment  required  choosing  network  topology, 
tracking  loop  bandwidth,  and  equipment  and  atmospheric  delay 
times  (if  reduced  phase  error  were  desired).  Once  network  para¬ 
meters  were  determined  and  each  node  programmed  to  begin  opera¬ 
tion,  timing  information  contained  in  exchanged  TRIP  data 
packages  allowed  all  nodes  to  run  at  the  same  frequency. 

In  summary,  it  is  essential  that  the  network  hierarchy  be 
predetermined  by  the  operator  and  that  each  chosen  slave  receives 
timing  information  over  a  single  link  in  such  a  way  that  all 
slaves  may  trace  a  reference  path  back  to  the  network  master. 

5.2.3  Mutual  Synchronization 

The  technique  of  Mutual  Synchronization  (MS)  is  one  in  which 
each  node  has  a  clock  that  may  be  adjusted  in  frequency  so  as  to 
reduce  the  timing  error  between  itself  and  some  average  of  the 
rest  of  the  network.  The  general  approach  calls  for  a  weighted 
sum  of  phase  errors  for  all  incoming  data  clocks  to  be  used  as  a 
control  signal  to  pull  the  node  clock  frequency.  The  network 
therefore  attempts  to  reach  a  stable  frequency  of  operation  using 
the  notions  of  feedback  control.  The  method  has  the  advantage 
that  removal  of  any  one  node  from  the  network  still  leaves  the 
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system  in  synchronism,  although  in  this  situation  and  others,  a 
transient  disturbance  will  propagate  around  the  network  until  a 
stable  equilibrium  frequency  has  been  reached. 

Each  operating  link  is  assigned  a  weighting  factor  between 
0  and  1,  and  all  clock  error  calculations  are  based  on  a  normali¬ 
zed  sum  of  all  link  weights  as  applied  to  calculated  clock  errors 
over  each  link.  Thus,  the  resulting  node  clock  error  contains 
information  from  each  operating  link.  This  technique  minimizes 
the  net  phase  difference.  Nodal  clocks  contribute  directly  to 
the  control  of  all  directly-connected  nodes  and  indirectly  to 
control  of  all  other  nodes  in  the  network.  A  highly  connected 
network  contains  many  feedback  paths  contributing  to  potential 
instability  because  each  node  in  a  duplex  communication  pair 
exhibits  some  control  on  the  opposite  side. 

It  seems  rather  obvious  that  Mutual  Sync  systems  are  in¬ 
herently  self-organizing  as  it  is  simply  a  matter  of  adjusting 
link  weighting  factors  upon  elimination  of  nodes  or  links  from 
the  network.  This  is  true  only  if  all  incoming  links  to  a  given 
node  are  equally  weighted,  thus  maintaining  balance  within  the 
network . 


5.2.4  Time  Reference  Distribution 

In  the  Time  Reference  Distribution  (TRD)  approach  to  network 
timing,  individual  node  clocks  are  subjected  to  occasional  cor¬ 
rection  using  timing  information  being  continually  passed  around 
the  network  from  node  to  node  [5.5].  In  this  regard,  there  are 
some  similarities  with  the  double-ended  frequency  averaging 
method  where  linked  nodes  exchanged  timing  data  via  a  service 
channel.  However,  the  Time  Reference  Distribution  system  goes 
well  beyond  this  concept.  Because  all  of  the  node  clocks  are 
ranked  in  an  order  of  increasing  importance,  the  network  has  the 
capability  of  locking  itself  totally  to  the  highest  ranked  clock 
currently  in  operation.  This  technique  [5.6]  has  been  discussed 
for  synchronization  of  the  Western  Union  digital  network  [5.7] 
and  the  DATRAN  [5.8]  network. 

The  Time  Reference  Distribution  scheme  offers  superior  ad¬ 
vantage  in  network  control  as  far  as  survivability  is  concerned. 

The  distinctive  feature  of  Time  Reference  Distribution 
is  that  a  network  with  reasonably  stable  clocks  at  each  node  Is 
forced  to  take  corrective  action  at  these  nodes  on  the  basis  of 
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network  time  references  that  are  continually  passed  around  the 
system.  All  nodal  clocks  are  rank-ordered,  and  the  timing  of 
each  Incoming  link  is  compared  with  the  local  clock  in  the  der¬ 
ivation  of  a  timing  error  for  that  link.  This  error  signal 
is  transmitted  back  to  the  originating  node  which  is  likewise 
computing  its  own  nodal  clock  error  for  the  link.  With  the  en¬ 
suing  exchange  of  clock  error  signals  between  both  ends  of  the 
link,  each  of  the  two  nodes  has  the  capability  of  computing  the 
true  nodal  clock  time  difference  with  virtually  no  dependence  on 
path  delay.  This  can  be  expressed  more  precisely  in  the  fol¬ 
lowing  way. 

Mathematically,  the  situation  can  best  be  described  by 
considering  the  transmission  from  two  nodes,  A  and  B,  of  a  time 
reference  pulse  (TRP) .  Node  B  measures  the  elapsed  time  of  the 
incoming  TRP  from  node  A  relative  to  its  own  clock,  resulting  in 
a  quantity  tA.  Similarly,  at  node  A  the  TRP  from  node  B  is  mea¬ 
sured  relative  to  the  local  clock  producing  tg.  Defining  the 
clock  TRP  emission  times  on  a  reference  time  scale  to  be  TA  and 
Tg ,  and  the  path  delay  from  A  to  B  as  TAg »  it  can  be  shown  that 


=  T.  +  r 


AB 


-  T, 


and 


-  tb  +  t 


BA 


-  T. 


Hence,  the  clock  difference,  (T^  -  TgJ,  can  be  computed  as: 


When  the  path  delay  in  both  directions  is  identical,  it  is 
apparent  that 

TA  "  TB  2(CA  "  Cb) 

and  once  the  measurements  tA  and  tg  are  available  at  both  nodes, 
the  clock  offset  can  be  calculated. 
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Apart  from  the  requirement  of  timing  error  exchange ,  the 
Time  Reference  Distribution  technique  functions  in  a  manner  which 
is  very  similar  to  that  of  hierarchical  Master/Slave.  The  con¬ 
trol  signals,  which  must  be  transferred  around  a  Time  Reference 
Distribution  controlled  network,  may  be  summarized  as  follows  for 
a  single  node: 

(1)  The  local  time  measurement  (t^)  for  a  particular 
incoming  link  (returned  to  the  originating  node) 

(2)  The  rank  of  the  node  being  used  as  a  master 
timing  node  for  the  local  node  (Dl) 

(3)  A  measure  of  the  path  quality  for  the  master 
node  supplying  the  local  clock  (D2) 

(4)  The  rank  of  the  local  node  clock  (D3) 

The  three  latter  items  are  used  with  a  set  of  prearranged  selec¬ 
tion  rules*  in  the  manner  described  earlier.  The  network  then 
locks  onto  the  timing  data  of  highest  rank. 

In  summary.  Time  Reference  Distribution  is  essentially  a 
directed  control  approach  with  additional  features  to  improve 
accuracy,  speed  at  which  a  network  can  bring  clocks  into  phase 
with  the  master,  and  a  greatly  enhanced  survivability  factor  due 
to  self-organizing  capabilities  and  clock  rank-ordered  topology. 
In  fact,  the  most  recent  change  to  the  Time  Reference  concept  is 
that  of  Phase  Reference  Combining  which  statistically  combines 
errors  from  all  links.  A  list  of  the  five  most  important  fea¬ 
tures  of  Improved  TRD,  or  most  desirable  characteristics,  for 
synchronous  networks  are: 

•  Directed  control 

•  Double-endedness 

•  Independence  of  clock  error  measurements 
and  corrections 

•  Self-organization 

•  Phase  reference  combining 


Selection  rules  are  discussed  in  Appendix  A. 
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5.3  Laboratory  Tests 

An  important  phase  of  the  in-house  development  schedule  was 
the  system  test  program.  With  simple  test  equipment  it  was  easy 
to  determine  the  performance  of  a  single  TS  or  a  small  network 
of  two  or  three  units .  The  ability  to  track  an  external  refer¬ 
ence  signal  without  TRIP  packages  is  a  very  useful  feature  of 
the  TS  hardware.  In  this  mode,  a  1-pps  signal,  applied  to  one 
of  the  auxiliary  reference  ports,  is  sufficient  to  enable  track¬ 
ing  of  the  signal  via  a  few  commands  from  the  operator.  This  is 
the  most  basic  mode  of  operation  of  the  Timing  Subsystem.  Auxi¬ 
liary  reference  operation  allows  several  very  simple,  yet  viseful, 
nodal  configurations.  First,  a  single  mode  can  be  instructed  to 
discipline  its  internal  clock  to  that  of  an  external  reference 
pulse.  Secondly,  a  master  node  can  be  configured  to  reference 
an  external  source  rather  than  its  own  clock,  thus  providing  net¬ 
work  synchronization  to  any  local  clock  at  the  master  node  site. 
Finally,  and  most  importantly,  an  external  time  reference,  such 
as  UTC  (NBS)  or  UTC  (USNO)  may  be  applied  to  the  Timing  Subsystem 
(after  1-pps  decoding  in  the  LORAN-C  or  satellite  receiver),  thus 
passing  universal  time  throughout  a  network  (directed  control 
only)  with  no  penalty  in  TS  hardware  complexity. 

Also  discussed  in  this  section  are  the  types  of  network 
level  experiments  run  in-house  used  to  observe  the  behavior  of 
the  candidate  timing  techniques  under  simulated  network  condi¬ 
tions.  Of  course,  all  data  paths  were  through  direct  wire  con¬ 
nections  to  each  TS.  However,  network  stress  could  easily  be 
introduced  through  the  operator's  terminal  or  by  physically 
removing  a  data  path  or  node. 

5.3.1  Auxiliary  Reference  Operation 

One  of  the  more  fundamental  measurement  capabilities  of  the 
Timing  Subsystem  is  the  technique  of  slaving  the  nodal  clock  to 
an  externally-applied  1-pps  reference.  Simply  stated,  this  pulse 
is  allowed  to  gate  the  time  interval  measurement  unit,  and  pro¬ 
cessed  errors  are  then  applied  to  the  software-controlled  digital 
Phase-Locked  Loop  (FLL)  for  clock  corrections.  First,  the  exter¬ 
nal  pulse  ( in  this  case  supplied  by  a  pulse  generator)  was  applied 
to  one  of  the  three  external  TRP  ports  located  at  the  rear  panel. 
Secondly,  a  performance  verification  facility  (Time  Interval 
Measurement  Counter)  was  connected  to  both  the  input  pulse  and 
the  adjusted  output  pulse  from  the  Timing  Subsystem.  Therefore, 
by  measuring  the  elapsed  time  between  these  pulses ,  one  is 
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effectively  measuring  the  error  signal  that  the  TS  is  attempting 
to  reduce.  Figure  5.2  shows  the  layout  of  the  test  setup.  Note 
that  there  are  three  Auxiliary  Time  Reference  Pulse  (AUXTRP) 
input  ports.  Connecting  the  reference  signal  to  any  one  of  the 
three  will  suffice  as  long  as  the  node  computer  is  instructed  as 
to  which  port  to  reference.  This  is  accomplished  by  the  appro¬ 
priate  command  at  the  operator's  terminal. 

If  desired,  the  operator  may  inspect  the  Performance  Assess¬ 
ment  (PA)  buffer  within  the  computer  memory  for  a  series  of 
averages  of  clock  error  and  the  average  difference  between  the 
TS  timing  source  and  each  of  the  three  external  reference  ports. 
This  information  is  available  from  the  operator's  keyboard. 

In  addition,  a  self-reference  mode  of  operation  can  be  spe¬ 
cified.  In  this  setup,  the  unadjusted  node  standard  1-pps  is 
applied  directly  to  one  of  the  auxiliary  reference  ports.  This 
effectively  allows  disciplining  of  the  nodal  reference  clock  to 
the  phase  of  the  external  frequency  source.  Again,  performance 
is  monitored  via  the  time  interval  counter  or  the  automatic  PA 
averaging  feature.  Figure  5.3  depicts  the  test  setup  for  self¬ 
reference  operation. 

Although  no  formal  performance  curves  are  available,  it  was 
determined  that  the  Timing  Subsystem  was  working  properly. 

Further  experimentation  involved  removing  the  reference  pulse 
and  observing  TS  behavior.  It  was  decided  at  this  point  that  a 
30-second  guard  time  would  be  incorporated  in  the  nodal  software 
that  would  disallow  any  major  clock  alterations  and  "coast" 

(using  previous  loop  output  history)  for  a  maximum  of  30  seconds 
if  no  error  measurement  data  were  available.  This  feature  pre¬ 
vents  the  TS  from  cycling  through  startup  and  acquisition  phases 
if  a  new  error  term  were  not  available  each  second.  This  is 
especially  important  when  confronted  with  a  high  error  rate. 
Remember  that  the  TS  utilizes  error- detection  hardware  but  is 
forced  to  discard  a  frame  when  the  errors  are  present.  Section 
6.1.4  recommends  error-correction  >ment  as  an  important  ele¬ 
ment  of  nodal  hardware  -  when  pass-  ming  and  status  infor¬ 
mation  in  a  single  frame  through  the  s.  vice  channel,  continuous 
errors  resulting  in  data  elimination  cannot  be  tolerated. 

5.3.2  Network  Operation 

The  first  attempt  at  network  synchronization  involved  a 
simple  Master/Slave  experiment  with  the  master  clock  disciplined 
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to  an  external  1-pps  source.  Although  no  formal  acquisition  and 
tracking  performance  data  were  taken,  it  was  clear  by  the  use  of 
the  interval  counter  that  the  two  subsystems  were  in  close  fre¬ 
quency  alignment. 

Startup  behavior  was  observed  very  carefully  for  possible 
overshoot,  inaccuracy,  or  measurement  foldover  error  conditions. 
The  coarse  delay  loop  implemented  in  software  was  capable  of 
stopping  the  node -adjustable  clock  over  a  range  of  4  ps  to  1  sec¬ 
ond  with  +4-JIS  accuracy.  To  eliminate  the  possibility  of 
erroneous  measurements,  clock  error  was  sampled  over  a  3-second 
interval  before  a  coarse  delay  adjustment  was  initiated.  The 
overall  stability  and  accuracy  of  the  coarse  delay  adjustment 
loop  can  be  attributed  to  the  use  of  a  multi-stage  software  loop 
technique  and  careful  calibration  (i.e.,  measurement)  of  instruc¬ 
tion  execution  times.  Once  the  coarse  error  adjustment  has 
brought  the  clock  to  within  4  ms  of  zero  error,  acquisition 
begins  and  the  phase  microstepper  is  instructed  to  slew  at 
100  ns/s  so  as  to  further  reduce  clock  error.  Tracking  is  ini¬ 
tiated  when  the  error  falls  within  a  programmable  tracking  region 
threshold.  For  typical  coarse  and  fine  thresholds  of  4  ms  and 
150  ns,  respectively,  the  time  to  reach  tracking  when  a  node 
enters  the  network  is  approximately  43  seconds  for  the  worst 
case;  but,  more  often,  times  in  the  15-25  second  range  were 
observed.  Figure  5.4  is  a  representative  curve  of  the  acquisi- 
tional  behavior  of  the  Timing  Subsystem.  Notice  that  startup 
mode  reduces  the  error  from  the  starting  point  (4  (is  *  1  s)  to 
+4  ms  within  three  seconds . 

Further  two-node  testing  included:  Master/ Slave,  both  with 
and  without  an  external  reference  for  the  master,  each  tested 
using  the  phase  microstepper  and  VCO  methods  of  clock  adjustment; 
Mutual  Synchronization;  and  Time  Reference  Distribution.  In  each 
case,  node  behavior  was  of  prime  importance  since  it  had  been 
sufficiently  demonstrated  that  the  software  PLL  was  working; 
i.e.,  the  Timing  Subsystem  was  certainly  able  to  acquire  and  lock 
onto  a  reference  signal  as  well  as  to  pass  timing  information 
over  wired  links.  Therefore,  the  in-house  network-level  testing 
was  focused  on  the  characteristics  of  TS  behavior  in  a  simulated 
network  environment  when  interacting  with  other  nodes. 

To  evaluate  nodal  performance  in  a  two-node  network,  a 
connection  scheme  was  devised  to  monitor  each  TS.  Figure  5.5 
shows  the  connections  for  a  single  link  two-node  performance 
evaluation  network.  Additionally,  a  three-node  delta  configu¬ 
ration  performance  evaluation  network  was  wired  to  monitor  all 
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Figure  5.4  Typical  Acquisition  Behavior  of  Timing  Subsystem 


three  TS's,  as  shown  in  Figure  5.4.  Notice  the  redundant  link 
between  TS2  and  TS3;  this  was  an  important  aspect  of  tests  in¬ 
volving  network  reorganization  features  (TRD) .  The  TS  is  forced 
to  choose  the  best  path  from  all  paths  to  receive  timing  infor¬ 
mation.  By  assigning  link  merits  (demerits)  for  each  communi¬ 
cations  path,  any  given  node  should  be  able  to  choose  the  path 
of  total  least  demerit  from  the  master. 

In  summary,  directed  control  (M/S,  MS,  TRD)  experiments 
performed  satisfactorily  as  expected.  Master/Slave  not  only 
was  the  simplest  timing  algorithm  to  implement  in  software  but 
also  was  the  easiest  to  set  up  for  an  experiment  in  a  network 
(except  Independent  Clock).  Master/Slave  was  always  capable  of 
frequency  synchronization  but,  including  equipment  delays  from 
back-to-back  measurements,  resulted  in  reduced  phase  error  as 
well.  Although  simple  Master/Slave  is  not  a  phase-coherent 
technique,  the  addition  of  phase -reference  combining,  or  other 
TRD-like  features,  would  further  enhance  M/S  performance  without 
greatly  complicating  matters.  On  the  other  hand,  Mutual  Synchro¬ 
nization  was  found  to  be  somewhat  cumbersome  and  unreliable  on 
the  whole.  Additional  error  signal  filtering  was  required  to 
decrease  the  possibility  of  an  unstable  network.  Laboratory 
tests  revealed  that  MS  was  somewhat  unpredictable  in  behavior 
and  a  problem  with  starting  up  the  network  was  observed.  Simply 
put,  how  do  nodes  just  entering  the  network  avoid  perturbing  the 
current  mutual  system?  How  do  two  nodes,  attempting  Mutual 
Synchronization,  begin  from  gross  clock  misalignment?  In 
attempting  to  overcome  these  problems,  less  confidence  was 
placed  in  this  technique  due  to  previously  mentioned  imple¬ 
mentation  difficulties.  For  this  and  other  reasons  described 
in  Section  6,  Mutual  Synchronization  was  considered  an  unsuit¬ 
able  timing  technique  for  DCS  use. 

Finally,  the  last  series  of  laboratory  tests  concerned  the 
interaction  of  three  nodes.  Master/Slave  experiments  demon¬ 
strated  that  directed  control  could  easily  align  clocks  in  fre¬ 
quency  through  several  levels  of  interconnected  nodes.  This  was 
observed  in  a  tandem  configuration  with  the  master  referencing  an 
external  frequency  source.  This  was  also  proved  on  a  delta-style 
network  with  a  redundant  link.  Since  each  slave  subsystem  was 
instructed  to  find  timing  information  on  a  single  predetermined 
link,  redundancy  or  closed  timing  loops  should  not  affect  the 
directed  control  path  from  master  to  slaves. 
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TIMING  SUBSYSTEM  (T  CLOCKS 


Figure  5.6  Three -Node  Performance  Evaluation  Network  (Delta) 


Laboratory  experiments  with  Time  Reference  Distribution 
(both  original  and  Improved  versions)  were  focused  primarily 
on  network  self -organization  features.  As  mentioned  earlier. 
Figure  5.6  shows  the  network  used  to  evaluate  self-organization 
experiments.  To  illustrate  the  tests  undertaken,  a  simple  ex¬ 
periment  was  conceived  to  exercise  the  hierarchy  rules  of  TRD 
as  interpreted  and  implemented  in  software  by  CNR.  For  this 
series  of  experiments,  the  original  Time  Reference  Distribution 

[5.5]  with  obsolete  data  handler  was  used.  As  far  as  network 
hierarchy  is  concerned.  Improved  Time  Reference  Distribution 

[5.6]  behaves  in  the  same  manner  except  link  qualities  are 
assessed  on  statistical  parameters  rather  than  on  integer  link 
demerits  found  in  the  original  version.  Following  Figure  5.7 
the  sequence  of  events  used  to  demonstrate  TRD  self-organization 
capabilities  were: 

Definition  of  Terms 

TS[n]  Timing  Subsystem  number  n 
L[n]  Link  number  n 

R=[n]  Node  clock  rank  n  (lower  numbers  = 

better  clocks) 

D=[n]  Link  demerit  (lower  numbers  = better  links) 

State  1:  Each  Timing  Subsystem  is  assigned  a  unique  clock  rank 
and  each  path  is  assigned  a  demerit.  Upon  reaching 
steady-state  network  operation,  the  bold  arrow(s)  show 
the  path  of  the  resultant  directed  control. 

State  2:  TS1  rank  was  degraded;  TS2  rank  was  improved.  A  switch 
of  network  master  to  TS2  was  observed.  Directed  con¬ 
trol  following  paths  of  least  demerit  to  subservient 
nodes  was  observed. 

State  3:  Link  3  demerit  was  raised  to  5.  A  resulting  path 

switch  was  observed  to  maintain  path  of  least  total 
demerit  to  nodes  lower  in  the  hierarchy. 

State  4:  TS3  rank  was  improved  so  as  to  be  the  best  in  the 

network.  Link  4  demerit  was  improved.  A  master  switch 
and  path  switch  were  observed,  as  indicated  on  the 
diagram. 

State  5:  Link  1  was  removed.  Link  3  demerit  was  improved. 

Link  4  demerit  was  degraded.  TS2  rank  was  improved 
so  as  to  be  the  best  in  the  network.  TS3  rank  was 
degraded.  Again,  the  algorithm  switched  master  to 
TS2.  Directed  control  path  was  over  links  of  least 
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demerit  so  as  to  maintain  lowest  total  path  demerit 
to  any  node  lower  in  the  hierarchy. 

State  6:  Links  3  and  4  were  removed.  TS1  rank  was  improved  to 
best  in  network.  TS2  was  removed  from  network  as  a 
result  of  eliminating  links  1,  4,  and  5.  Path  switch 
and  master  switch  observed  according  to  TRD  hierarchy 
rules . 

Clearly,  the  TRD  hierarchy  experiments  were  successful. 
Although  long-term  stability  and  accuracy  data  were  not  acquired, 
it  seemed  that  TRD  performed  as  well  as  simple  Master/Slave. 
Additional  long-term  tests  were  not  possible  due  to  time 
limitations.  However,  it  seemed  certain  that  TRD  would  func¬ 
tion  properly  in  the  field. 

5.4  Performance  Assessment 


It  was  evident  from  the  outset  of  the  field  test  program 
that  a  conscientiously-applied  performance  measurement  technique 
was  as  important  as  the  tests  themselves;  that  is,  verifying  that 
a  given  node  is  transmitting  and  receiving  data  over  a  link  is  no 
guarantee  that  synchronization  exists  at  the  network  level. 
Through  a  redundant  set  of  connections  to  the  Frequency  Measure¬ 
ment  Terminal  and  a  LORAN-C  receiver  at  each  site,  a  performance 
measurement  system  was  configured  to  enable  assessment  of  syn¬ 
chronization  performance  at  both  nodal  and  network  levels. 
References  [5.9]  and  [5.10]  provide  further  insight  into  the 
use  of  universal  time  as  a  network  timing  source. 

5.4.1  Performance  Measurement  System 

The  basic  function  of  the  Performance  Measurement  system  is 
to  record  averages  of  time  interval  measurements  from  various 
interconnected  devices.  The  measurement  system  consisted  of  the 
following  devices  located  at  each  site:  Timing  Subsystem,  Fre¬ 
quency  Measurement  Terminal  with  LORAN-C  receiver,  and  a  fre¬ 
quency  standard.  Figure  5.8  shows  how  this  equipment  was  wired. 

The  Frequency  Measurement  Terminal  should  be  considered  an 
integral  part  of  a  network  timing  experiment.  It  not  only  pro¬ 
vides  time  interval  measurement  capabilities  for  up  to  three 
external  clocks  but  also  houses  a  remote  telephone  data  coupler 
which  can  be  used  to  query  the  Timing  Subsystem  from  a  distant 
location.  Each  Frequency  Measurement  Terminal  (FMT)  also  con¬ 
tains  a  LORAN-C  receiver  whose  1-pps  output  is  typically 
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connected  to  measurement  channel  C  on  the  central  processor. 
Comparing  LORAN-C  readings  with  the  applied  l-pps  source  clock 
to  the  FMT  results  In  a  plottable  drift  rate  which  can  easily  be 
compared  to  LORAN-C  readings  at  NBS  and,  ultimately,  the  U.S. 
National  Time  Standard.  Through  these  means,  any  external  clock 
applied  to  an  FMT  housing  an  integral  LORAN-C  receiver  may  be 
compared  rather  easily  with  any  other  time  source  directly  con¬ 
nected  to  any  other  FMT  unit. 

For  detailed  instructions  on  setup  conditions  and  operation 
of  the  FMT,  the  reader  is  asked  to  consult  [5.11]. 

The  performance  measurement  system  gathers  time  interval 
measurement  averages  from  both  the  FMT  and  Timing  Subsystem. 

The  measurement  capabilities  are  briefly  described  below. 

•  A  rubidium  frequency  standard,  located  at  each  site,  was 
considered  the  node  frequency  standard  and  was  used  to 
provide  5-MHz  and  l-pps  references  for  both  the  FMT  and 
TS;  that  is,  all  FMT  measurements  are  elapsed  time  from 
the  frequency  standard  pulse  (1  pps)  to  the  pulse  being 
measured.  The  TS  uses  the  5-MHz  reference  for  a  timing 
source  and  the  l-pps  reference  to  compare  against  the 
node  adjustable  clock. 

•  The  FMT  can  directly  compare  the  decoded  LORAN-C  pulse 
with  that  of  the  frequency  standard,  thus  providing  a 
path  to  an  external  reference  (Seneca,  NY,  LORAN-C 
station  with  cesium  beam  standard)  to  which  all  nodes 
may  be  compared.  This  is  the  method  by  which  universal 
time  may  be  introduced  into  the  network. 

•  Redundancy  exists  between  TS  and  FMT  measurements  of 
LORAN-C  and  frequency  standard  pulses  to  further  rein¬ 
force  performance  curves.  In  more  detail,  the  FMT 
measures  elapsed  time  between  the  frequency  standard 
l-pps  and  LORAN-C  as  well  as  frequency  standard  vs.  TS 
adjusted  l-pps  output.  Similarly,  the  TS  measures 
elapsed  time  between  its  adjusted  l-pps  output  and 

the  frequency  standard  l-pps  as  well  as  TS  vs.  LORAN-C. 

•  The  FMT  keeps  continuous  running  averages  over  4-hour- 
intervals,  each  consisting  of  200  samples. 

•  The  Timing  Subsystem  saves  24  hours  of  10-minute  averages 
of  600  samples  each  in  volatile  memory  that  is  available 
to  the  operator. 


5.4.2  Methods  of  Verifying  Network  Performance 

Once  a  timing  experiment  has  begun ,  all  data  logging  is 
automatic.  The  Timing  Subsystem  has  the  capability  to  store 
10-minute  averages  of  600  samples  each  for  24  hours.  Each 
sample  average  contains: 

(1)  Time-of-day  at  end  of  sample  period 

(2)  Average  nodal  clock  error  (as  calculated) 

(3)  Average  of  node  clock  vs.  AUXTRP1 

(4)  Average  of  node  clock  vs.  AUXTRP2 

(5)  Average  of  node  clock  vs.  AUXTRP3 

It  is  obvious  that,  in  order  to  find  the  drift  rate  between 
the  node  adjustable  clock  and  an  external  clock,  it  is  necessary 
to  connect  the  external  clock  1-pps  output  to  one  of  the  three 
external  TRP  (AUXTRP)  ports .  With  this  connection  and  a  Fre¬ 
quency  Measurement  Terminal  at  each  site,  the  user  has  the  faci¬ 
lities  to  compare  an  external  local  clock  to  any  other  in  the 
network  as  well  as  to  the  U.S.  National  Time  Standard  through 
LORAN-C  or  TV  Line-10  readings  at  remote  FMT's. 

A  Frequency  Measurement  Terminal,  located  at  the  National 
Bureau  of  Standards,  Boulder,  Colorado,  is  driven  by  the  U.S. 
National  Time  Standard.  This  terminal  also  receives  trans¬ 
missions  from  the  LORAN-C  transmitter  stations  at  Dana,  Indiana, 
Fallon,  Nevada,  and  from  each  of  the  three  network-affiliated 
television  stations  in  their  local  area. 

The  drift  rate  (At/T)  of  the  clocks  used  to  drive  the  FMT's, 
located  at  the  RADC  sites  relative  to  the  U.S.  National  Bureau 
of  Standards  in  Boulder,  Colorado,  can  be  measured  as  follows: 

•  Put  the  Local  Frequency  Measurement  Terminal  in  the 
telephone  mode. 

•  Call  the  Frequency  Measurement  Terminal  located  at 
the  National  Bureau  of  Standards,  Boulder,  Colorado. 

•  Read  out  the  eight  days  of  stored  data  from  the 
LORAN-C  channel  (C)  and  each  of  the  network  tele¬ 
vision  channels.  Procedure  is  described  in  [5.11]. 

•  Compare  the  time  difference  readings  at  NBS  with  the 
time  difference  readings  taken  at  the  local  terminal 
for  the  LORAN-C  data  and  data  from  each  of  the  three 
network  television  channels  (the  LORAN-C  data  is  far 
more  accurate  than  the  TV  data) . 


•  Plot  the  drift  rate  (At/T)  for  the  8-day  period. 

Continue  plotting  this  data  for  subsequent  periods 
as  long  as  it  is  necessary  to  satisfy  local 
requirements . 

The  Frequency  Measurement  Terminals  will  accept  one  pulse- 
per-second  outputs  from  up  to  three  additional  clocks  and  will 
measure,  identify,  and  record  minutely  time  differences,  averaged 
over  4**hour  periods,  between  the  clock  in  the  terminal  and  each 
of  the  three  external  clocks.  Drift  rates  (At/T)  between  the 
clock  in  the  terminal  and  the  clock  being  calibrated  can  be  cal¬ 
culated  for  as  long  a  period  as  is  necessary  to  satisfy  local 
requirements . 

Delay  may  be  added  (subtracted)  from  the  external  clock 
being  calibrated  until  the  time  difference  between  it  and  the 
clock  in  the  FMT  is  zero  plus  or  minus  any  time  error  in  the 
terminal  clock  relative  to  the  USNO  or  NBS  master  clocks  as 
maintained  in  the  log  for  the  clock  in  the  terminal. 

Drift  rate  (At/T)  can  be  calibrated  for  the  external  clock 
using  the  procedures  outlined  above.  These  procedures  will  cali¬ 
brate  the  drift  rate  of  the  external  clock  relative  to  the  clock 
in  the  terminal.  To  calibrate  the  external  clock  relative  to  the 
USNO  or  NBS  master  clocks,  a  second  drift  rate  (At/T)'  must  be 
calculated  using  the  measured  drift  rate  data  of  the  external 
clock  relative  to  the  terminal  clock  and  the  measured  drift  rate 
data  of  the  terminal  clock  relative  to  the  USNO  or  NBS  master 
clocks.  This  second  drift  rate  (At/T)'  calculation  is  then  used 
to  calibrate  the  external  clock,  as  outlined  earlier. 

5.5  Field  Tests 


Following  a  brief  review  of  the  manner  in  which  performance 
data  was  gathered,  a  section  is  devoted  to  each  of  the  timing  tech¬ 
nique  experiments  performed  on  the  single-ended  tandem  network 
described  in  Section  3.  It  was  not  feasible  to  exercise  each  candi¬ 
date  timing  technique  so  emphasis  was  placed  on  the  frequency 
alignment  (directed  control)  techniques.  No  experiments  were  run 
using  Independent  Clock,  Mutual  Synchronization,  or  Improved  Time 
Reference  Distribution  candidates  for  reasons  previously  discussed. 

Included  with  a  description  of  each  network  test  is  a 
diagram  of  the  network  used  and  a  brief  evaluation  of  timing 
technique  performance.  Performance  curves  depict  the  frequency 
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offset  of  each  node  as  compared  to  the  Seneca,  NY,  LORAN-C  Test 
Bed  Master.  Specific  difficulties  and  obstacles  affecting  the 
measurement  process  are  noted  where  applicable. 

One  point  worth  noting  concerns  the  node  setup  and  initiali¬ 
zation  parameters.  No  attempt  was  made  to  experiment  with 
various  clock  control  loop  parameters.  Additionally,  although 
equipment  delays  were  measured  and  path  delays  were  known  from 
published  material,  no  attempt  was  made  to  use  the  data  in  the 
TS  as  a  means  to  minimize  clock  phase  offset  between  nodes  in  a 
single-ended  measurement.  Also,  no  precise  time  experiments  were 
undertaken.  However,  it  would  not  be  difficult  to  use  equipment 
and  path  delay  quantities  to  reduce  phase  error  -  merely  a  tedious 
task  and  not  of  prime  importance  for  thi6  experiment  series. 

A  tabularized  summary  of  parameters  used  in  the  network 
experiments  will  be  presented  in  Section  5.6. 

5.5.1  Two -Node  Mas ter /Slave  (TROPO) 

A  simple  single-ended  directed-control  test  was  performed 
over  the  168-mile  TROPO  range  in  the  Youngstown-to-Verona 
direction.  A  diagram  of  the  equipment  used  at  each  site  is 
shown  in  Figure  5.9.  Quad  diversity  was  used  to  maximize  recep¬ 
tion  levels  and  minimize  bit  errors. 

Almost  immediately  it  was  observed  that  even  with  a  strong 
received  signal  level  and  the  normally  moderate  TROPO  Bit  Error 
Rate  of  about  10  °  [5.2],  the  Timing  Subsystem  was  rejecting 
TRIP'S  at  about  one  per  five  or  six  frames  received.  Although 
this  observation  was  not  surprising,  it  emphasizes  the  importance 
of  error-correction  equipment  for  error-free  TROPO  reception. 

This  problem  is  further  compounded  if  a  double-ended  transfer 
was  attempted.  Double-ended  clock  error  calculations  require 
buffering  of  frames  from  the  current  and  the  previous  exchange 
at  each  node,  thus  quadrupling  the  error-free  TRIP  requirements 
over  single-ended  transfers. 

Careful  monitoring  of  TS  status  through  the  debugging 
capabilities  of  the  TS  software  revealed  that  TROPO  fading  was 
perturbing  the  bit  stream  to  the  point  where  frame  boundary 
flags  were  not  being  received,  thus  causing  frame  over-runs  and, 
often  times,  receiver  aborts.  Moreover,  these  "endless"  frames 
were  overwriting  TRIP  memory  storage  locations  and  destroying 
data  from  other  channels  -  a  totally  unacceptable  condition. 
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The  cure  for  this  problem  involved  a  change  in  software  for  the 
Receive  Link  Termination  Processors  (RLTP) .  The  Computer  Program 
Manual  for  Timing  Subsystem  Development  [5.4]  describes  in  detail, 
with  accompanying  flowcharts ,  the  RLTP  processing  scheme  for  in¬ 
coming  data.  Incorporated  was  a  byte-count  check  for  each 
incoming  byte  or  byte  pair.  If  the  byte  count  did  not  exceed 
the  predetermined  frame  length  (not  including  flags  or  CRC  check 
bits),  then  the  received  bytes  were  committed  to  memory.  If  the 
byte  count  was  exceeded  before  receiving  a  closing  flag,  then  the 
RLTP  was  automatically  aborted,  the  TRIP  ignored,  and  the  search 
for  a  new  opening  flag  was  begun.  Implementation  of  this  check 
resolved  the  problem  caused  by  the  characteristics  of  the  fading 
channel.  As  an  aid  to  development,  the  TRIP  frame  length  (in 
bytes)  was  kept  as  a  programmable  feature,  thus  allowing  a 
variety  of  frame  lengths  to  be  chosen. 

For  a  look  at  typical  TROPO  received  signal  levels  for  C- 
band  quad  diversity,  refer  to  Figure  5.10.  Notice  that  fairly 
large  signal  level  excursions  are  observed  at  about  an  8-10  Hz 
rate  and  that  relatively  deep  fades  occur  about  one  to  two  times 
per  second.  This  chart  is  from  the  June  1981  series  of 
experiments . 

Summarizing  Timing  Subsystem  performance  over  TROPO  links 
may  be  simply  stated:  For  truly  reliable  performance,  error- 
correction  equipment  is  required.  This  is  especially  true  since 
both  timing  and  status  information  are  sent  and  received  over  the 
service  channel  (option  C,  Section  5. 1.2. 3  of  [5.2]).  However, 
there  is  no  doubt  that  a  fading  dispersive  channel  can  support 
a  timing  function  with  very  acceptable  accuracy. 

This  experiment  duration  was  approximately  28  hours.  With  the 
Youngstown  node  tracking  the  local  rubidium  frequency  standard  and 
the  frequency  difference  between  the  site  rubidium  and  the  re¬ 
ceived  LORAN-C  from  Seneca  known  a-priori,  it  was  a  simple  matter 
to  plot  the  performance  of  the  master  and  slave  clocks  relative 
to  a  single  reference.  Figure  5.11  shows  the  relative  frequency 
rates  of  the  three  locations.  Seneca,  assumed  to  be  the  Test  Bed 
Master,  is  the  baseline  and  has  zero  frequency  offset.  The  cumu¬ 
lative  error  of  the  sites  shows  up  as  a  slight  frequency  dif¬ 
ference  as  compared  to  Seneca.  However,  it  does  not  matter  what 
that  offset  is  as  long  as  both  master  and  slave  exhibit  zero 
relative  frequency  offset.  Without  exceeding  the  measurement 
capabilities,  it  was  determined  that  this  Master/Slave  experiment 
was  performing  satisfactorily.  In  fact,  the  slight  frequency 
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difference  between  the  sites  is  probably  due  to  cumulative 
roundoff  errors  in  the  sample  averages  and  the  linear  regression 
process.  No  attempt  was  made  to  detail  short-term  fluctuations 
and  perturbations  because  many  of  these  effects  are  lost  in 
longer-term  averages  computed  by  both  the  TS  and  the  FMT.  As 
a  result,  we  wish  only  to  point  out  the  overall  trends  of  the 
nodes  as  compared  to  an  outside  reference.  With  this  in  mind, 
Master/Slave  over  TROPO  links  results  in  very  acceptable  fre¬ 
quency  alignment. 

5.5.2  Two-Node  Master/Slave  (LOS) 

A  similar  Master/Slave  experiment  was  conducted  over  the 
LOS  microwave  link.  This  was  a  single-ended  experiment  with 
Verona  as  master  and  GAFB  as  slave.  Figure  5.12  shows  the  net¬ 
work  equipment  connections.  Run  duration  was  approximately 
40  hours  at  which  point  power  was  lost  at  Verona  and  an  indefi¬ 
nite  coast  was  observed  at  the  slave  site  (GAFB).  This  behavior 
would  be  typical  of  that  of  a  stressed  network.  When  the  master 
is  lost,  all  subservient  nodes  must  rely  on  past  clock  correction 
history  (retained  in  PLL  memory)  rather  than  ceasing  operation. 
This  usually  results  in  better  performance  than  no  clock  correc¬ 
tions  at  all. 

Figure  5.13  depicts  the  relative  frequency  of  each  site 
compared  to  the  Seneca  reference.  The  master  clock  (Verona) 
was  disciplined  to  follow  the  site  rubidium  standard.  This  was 
a  logical  choice  because  the  history  of  the  rubidium's  perfor¬ 
mance  compared  to  Seneca  was  already  known,  so  its  drift  could 
be  predicted  after  power  was  lost.  The  frequency  plots  are 
purposely  smoothed  by  regression  to  better  visualize  the  longer- 
term  trends.  Unfortunately,  time  did  not  permit  longer  run 
durations  but,  clearly,  the  experiment  was  a  success  and  satis¬ 
factory  frequency  alignment  was  obtained. 

5.5.3  Three-Node  Tandem  Master/Slave 

A  single-ended  tandem  network  was  configured  according  to 
the  diagram  in  Figure  5.14.  With  Youngstown  as  master,  timing 
control  was  in  the  Youngstown-Verona-GAFB  direction.  Expected 
performance  would  be  the  following:  Each  slave  node  should  dis¬ 
cipline  its  clock  so  that  the  sum  of  the  frequency  offset  ^f  the 
rubidium  standard  plus  that  of  the  Timing  Subsystem  should  equal 
the  frequency  offset  of  the  Youngstown  rubidium  (all  referenced 
to  the  Seneca  LORAN-C  pulses).  This  was  observed  during  a 


Figure  5.14  Three-Node  Tandem  Network  with  LOS  and  TROPO  Links 


32-hour  run,  the  results  of  which  are  plotted  in  Figure  5.15. 

It  is  evident  that  each  of  the  slave  nodes  had  sufficiently 
altered  the  frequency  of  the  site  standard  so  that  they  were 
closely  following  the  master's  frequency.  No  detectable  fre¬ 
quency  offset  was  found,  at  least  to  the  limits  of  the  measure¬ 
ment  facilities  (about  one  part  in  10°  on  a  per-second  basis). 

Both  slave  nodes  closely  followed  the  frequency  offset  of  the 
Youngstown  master  as  compared  to  the  Seneca  LORAN-C  transmissions. 
This  plot  merely  presents  the  frequency  trends  of  each  node;  it 
is  not  intended  to  detail  small  fluctuations  in  clock  frequency. 
The  primary  reason  for  this  smoothing  is  dictated  by  the 
averaging  techniques  used  by  the  Timing  Subsystem  and  the 
Frequency  Measurement  Terminal;  that  is,  short-term  fluctuations 
in  clock  adjustments  are  masked  by  the  averaging  which  combines 
hundreds  of  samples  over  a  known  interval.  Essentially,  a  long¬ 
term  (four  hours)  average,  as  used  in  the  FMT,  is  suitable  for 
comparing  long-term  frequency  offsets. 

5.5.4  Three-Node  Tandem  Time  Reference  Distribution 

The  TRD  experiment  demonstrated  the  performance  of  the 
original  TRD  technique  as  described  in  [5.5],  Since  the  Improved 
TRD  technique  was  meant  to  be  used  with  double-ended  transfers, 
it  was  not  attempted;  however,  indications  are  that  it  would 
behave  similarly  to  TRD  with  perhaps  small  improvements  in  phase 
accuracy.  The  Test  Bed  measurement  facilities  were  not  equipped 
to  measure  node-to-node  precise  phase  conveniently. 

Each  node  in  the  tandem  network  was  assigned  a  unique  r&nk 
with  Youngstown  being  the  highest  ranking  clock,  Verona  second, 
and  GAFB  third.  This  was  not  an  arbitrary  assignment  because  the 
use  of  single-ended  transfers  dictated  the  direction  of  control, 
being  from  Youngstown  to  Verona  to  GAFB.  In  addition,  the  trans¬ 
mission  links  were  assigned  a  demerit  rating  with  the  LOS  link 
being  rated  about  five  times  better  than  the  TROPO  link.  Both 
clock  and  link  ratings  are  integer  quantities  so  that  choosing 
a  specific  value  is  secondary  with  respect  to  choosing  a 
precedence. 

With  the  node  parameters  determined,  the  network  was  in¬ 
structed  to  begin  operation.  It  was  immediately  observed, 
through  inspection  of  incoming  link  data,  that  a  network  hier¬ 
archy  was  formed  in  just  several  iterations  (e.g.,  several 
seconds).  Once  in  steady-state  operation,  the  network  behaved 
identically  to  the  three-node  Master/Slave  experiment  just 
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described.  It  would  be  expected  that  TRD  should  have  improved 
accuracy  over  simple  M/S  (by  virtue  of  combining  the  computed 
error  of  the  node's  neighbor  with  its  own),  but  this  was  not 
detectable  with  the  available  performance  assessment  equipment. 
Certainly,  as  shown  in  the  plot  in  Figure  5.16,  three-node  tandem 
TRD  performed  at  least  as  well  as  the  similar  M/S  experiment. 

Each  slave  node  had  adjusted  its  clock  to  be  equal  in  frequency 
to  that  of  the  network  master  (Youngstown)  node. 

5.5.5  Three-Node  Tandem  Master/Slave;  Master  with 
Large  Frequency  Offset 

Perhaps  the  most  demanding  experiment  in  terms  of  clock 
correction  capability,  three-node  tandem  Master/ Slave  with  the 
master  incorporating  a  large  frequency  offset,  demonstrated  net¬ 
work  behavior  under  slightly  less  than  ideal  conditions.  The 
network  configuration  was  identical  to  that  pictured  in  Figure 
5.14  and  nodal  parameters  identical  to  the  first  tandem  M/S 
experiment  described  in  Section  5.5.3.  In  addition,  the  Youngs¬ 
town  node  (master)  was  instructed  to  slew  its  clock  at 
2.7994  ns/s  (arbitrary)  through  the  phase  microstepper.  This 
added  frequency  offset  would  enhance  measurements  by  moving 
away  from  the  zero  frequency  offset  region.  Remember  that 
Seneca  was  chosen  as  the  Test  Bed  master  and  that  each  site's 
rubidium  standard  frequency  offset  was  known  a-priori.  The 
extra  offset  will  make  the  frequency  differences  between  each 
site's  rubidium  standard  and  Seneca  more  pronounced,  thus  faci¬ 
litating  performance  assessment  procedures. 

Figure  5.17  shows  the  frequency  of  each  node  (obviously 
identical)  compared  with  the  Seneca  LORAN-C.  Note  the  change 
of  scale  on  the  vertical  time  difference  axis.  This  shows  quite 
a  substantial  (about  three  parts  in  10^)  frequency  offset  from 
the  normally  free-running  frequency  standard  at  Youngstown.  The 
experiment  demonstrated  that  each  slave  node  also  adjusted  the 
frequency  of  its  clock  to  match  that  of  the  master.  Note  that 
the  maximum  adjustment  range  for  the  phase  microsteppers  used 
in  this  experiment  is  +100  ns/s,  or  one  part  in  10'  per  second; 
that  is,  the  device  is  capable  of  handling  quite  a  large  fre¬ 
quency  range . 
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5.5.6  Three-Node  Tandem  Master/Slave  with  Slave  VCO 

This  final  experiment  called  for  the  use  of  an  adjustable 
VCO  for  slave  clock  alignment  instead  of  the  rubidium  frequency 
standard  and  phase  microstepper  combination.  The  VCO  was 
selected  as  an  option  and  a  cost-reducing  method  of  adjusting 
the  node  frequency.  The  VCO  and  associated  D/A  hardware  are 
contained  within  the  Timing  Subsystem.  Clock  corrections  were 
made  via  an  algorithm  relating  the  adjustment  voltage  to  the 
resulting  frequency  change.  Two  Timing  Subsystems  were  built 
with  VCO  hardware  included,  but  only  the  GAFB  machine  was  con¬ 
figured  as  a  slave  VCO  in  the  tandem  M/S  experiment.  Nodal 
software  accommodated  both  methods  all  of  the  time  by  sending 
control  words  to  both  phase  microstepper  and  VCO  ports  whether 
they  were  connected  or  not. 

Before  presenting  a  brief  comparison  of  the  two  clock  ad¬ 
justment  techniques,  refer  to  Figure  5.18  for  the  performance 
curves  for  this  experiment.  Note  that  each  node  again  has 
tracked  the  master  (Youngstown)  frequency,  including  the  large 
offset  dialed  in  for  the  previous  experiment.  It  seems  that  the 
VCO  performed  admirably  even  with  a  large  frequency  difference 
to  correct  for.  Noticeable  short-term  variations  were  observed 
by  connecting  a  scope  to  the  node  adjusted  and  unadjusted  fre¬ 
quencies  and  comparing  the  relative  drift  between  them.  Actually, 
the  short-term  instability  can  be  readily  accounted  for  although 
we  did  not  attempt  to  plot  them  because  of  the  inherent  uncer¬ 
tainty  from  the  long  averages  employed  by  the  performance 
measurement  system.  Overall,  the  frequency  trends  of  each  node 
are  quite  in  line  with  the  expected  results. 

To  clarify  the  mechanism  responsible  for  short-term  VCO 
variations,  refer  to  the  frequency  adjustment  range  plot 
(Figure  5.19)  of  a  typical  VCO  used  in  the  TS  design.  Notice 
that  the  transfer  is  nonlinear  overall,  but  may  be  considered 
piecewise  linear  between  any  two  points  within,  say,  one  volt 
of  each  other.  Ideally,  to  maximize  the  accuracy  of  the  clock 
adjustment  scheme,  the  frequency  vs.  voltage  characteristic 
should  be  carefully  measured  and  plotted,  a  transfer  function 
derived,  and  a  "smart"  software  driver  designed  to  provide  con¬ 
trol  from  given  error  calculations.  However,  no  two  VCO  crystals 
exhibit  exactly  the  same  transfer  characteristics  or,  for  that 
matter,  have  the  same  range  or  center  frequency.  Therefore,  at 
best,  implementing  the  VCO  control  software  is  a  compromise;  but, 
as  described  earlier,  satisfactory  performance  can  be  achieved 
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FREQUENCY  OFFSET  (CENTER- 5  MHz) 


Figure  5.19 
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Useful  Adjustment  Range  of  Voltage-Controlled 
Oscillator 


at  minimal  cost.  Actually,  it  was  decided  to  describe  the  VCO 
transfer  characteristics  as  two  linear  segments  of  different 
slopes,  intersecting  at  the  center  frequency,  as  shown  by  the 
dotted  lines  in  Figure  5.19.  Actual  error  was  maximum  at  the 
midpoint  of  each  of  these  segments .  One  of  the  primary  reasons 
for  describing  the  VCO  characteristics  this  way  was  to  minimize 
software  overhead.  It  was  imagined  that  most  of  the  corrections 
would  be  at,  or  near,  the  center  frequency,  thus  reducing  the 
computational  burden  and  eliminating  much  of  the  interpolation 
process  needed  when  off-center.  Also,  frequency  corrections  are 
made  on  a  relative  basis  to  avoid  glitches  caused  if  the  computer 
were  to  set  the  VCO  by  starting  at  either  boundary  (for  a  base¬ 
line  reference)  and  stepping  the  desired  number  of  counts.  The 
relative  move  technique  insured  smooth  adjustments  each  itera¬ 
tion.  Overall,  the  approach  and  implementation  were  selected  to 
reduce  software  overhead  while  maintaining  a  satisfactory  level 
of  performance.  In  that  light,  it  was  a  successful  endeavor. 

Table  5-2  highlights  some  of  the  features  of  both  the  phase 
microstepper  and  VCO  methods  of  clock  adjustment.  Notice  that 
there  is  a  significant  cost  differential  between  the  devices  and 
that  the  microstepper  requires  an  external  frequency  source  as 
well.  This  factor  alone  is  of  enough  importance  to  lead  us  to 
the  recommendation  (see  Section  6)  that  VCO's  be  used  at  all 
less  important  nodes,  or  at  least  at  sites  where  a  frequency 
standard  is  not  already  present. 

5.6  Interpretation  of  Test  Results 

In  this  section  we  summarize  the  findings  of  each  of  the 
six  experiments  held  with  two-node  and  three-node  networks. 

A  brief  account  of  the  methodology  used  to  determine  network 
performance  is  also  described.  Included  are  several  tables: 
a  node  parameter  summary  for  each  experiment;  and  a  summary  of 
frequency  offsets  of  each  node  compared  to  the  Test  Bed  master 
1  pps  from  Seneca.  The  latter  table  discloses  the  undisciplined 
(free -running)  frequency  offset  of  each  site  standard  compared 
to  Seneca,  as  well  as  disciplined  clock  frequency  offsets  during 
experiment  runs . 

Armed  with  this  information,  the  reader  should  review  the 
performance  curves  in  Section  5.5  and  compare  disciplined  and 
undisciplined  clock  frequency  offsets  relative  to  the  Test  Bed 
master.  An  example  of  an  experiment  evaluation  session  high¬ 
lights  Section  5.6.1. 
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TABLE  5-2 


COMPARISON  OF  CLOCK  ADJUSTMENT  TECHNIQUES 


Specification 

Phase 

Microstepper* 

Voltage-Controlled 

Oscillator** 

i 

,  Resolution 

1  x  10“13 

-lit 

6.4x10 

Accuracy 

-q 

1x10  * 

. 

Maximum 

Slew  Rate 

+100  ns/s 

+160  ns/s  ! 

i 

1 

Total  Dynamic 
Range 

lx  106 

i 

2.5  xlO3 

1  l 

Control  Word 
Format 

6 -Decade  BCD 

12 -Bit  Binary 

CPU  Control 
Method 

Parallel  1/0 
to  Phase 
Microstepper 

Parallel  I/O  to 

D/A  Converter 
to  Oscillator 

!  Cost 

$  3500ft 

Approx.  $,  500+ 

NOTES : 

Phase  Microstepper  is  Austron  Model  2055A 

Voltage-Controlled  Oscillator  is  Vectron  Model  CO-204WV 

*  12-bit  D/A  conversion  for  control  voltage 
1 1 

Requires  external  frequency  source 
+Requires  D/A  conversion  circuitry  for  CPU  control 

5-45 


Finally,  Section  5.6  closes  with  a  brief  review  of  time 
transfer  requirements  (including  clock  error  and  path  delay 
calculations)  for  network  synchronization. 

5.6.1  Summary  of  Field  Tests 

In  order  to  simplify  the  presentation  of  the  network  experi¬ 
ment  series  described  in  Section  5.5,  the  reader  is  referred  to 
several  tables  and  diagrams.  Table  5-3  contains  a  glossary  of 
terms  used  in  describing  network  configurations  and  parameters. 
Note  that  the  use  of  the  VCO  implies  that  the  frequency  source 
is  contained  with  the  Timing  Subsystem,  and  no  external  reference 
need  be  applied.  Performance  measurement  is  carried  out  as 
normal  using  the  site  rubidium  standard  as  the  time  interval 
measurement  driver  for  the  FMT.  The  VCO  was  used  only  on  the 
GAFB  subsystem.  All  others  used  an  external  5-MHz  reference 
supplied  by  the  site  rubidium  standard. 

Figure  5.20  summarizes  the  network  configurations  for  each 
experiment.  These  diagrams  show  the  nodal  frequency  sources, 
clock  adjustment  technique,  and  the  path  of  directed  control. 

For  the  TRD  experiment,  clock  ranks  and  link  demerits  are  also 
shown. 

A  list  of  nodal  and  network  parameters  are  located  in 
Table  5-4.  Notice  the  delay  figures  listed.  These  are  based 
on  actual  equipment  delay  measurements,  but  their  use  in  fre¬ 
quency  alignment  experiments  is  optional.  This  is  the  case  for 
all  of  the  experiments  conducted  at  RADC.  TRD  is  capable  of 
phase  tracking,  but  it  was  not  evaluated  in  this  manner. 

The  Seneca,  NY,  LORAN-C  station  was  considered  to  be  the 
Test  Bed  master.  This  station  is  a  slave  to  the  Dana,  IN,  east 
coast  chain  master.  Since  experiments  were  concerned  with 
frequency  alignment,  the  first  step  to  calibrate  the  network  was 
to  measure  the  frequency  offset  of  each  of  the  three  site  rubi¬ 
dium  standards  and  to  compare  them  to  the  cesium  beam  standard 
at  Seneca.  Through  the  Frequency  Measurement  Terminals,  located 
at  Youngstown,  Verona,  and  Griff iss  AFB,  we  were  able  to  plot 
long-term  frequency  offset  for  each  rubidium  standard.  The  stan¬ 
dard  was  used  as  a  driving  (source)  1  pps  in  each  case,  thus 
providing  a  direct  comparison  of  received,  LORAN-C,  and  rubidium 
outputs.  The  results  of  this  measurement  series  is  plotted  in 
Figure  5.21.  Notice  that  data  was  gathered  over  a  period  of 
about  eight  days  through  averages  from  the  FMT.  With  Seneca 
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TABLE  5-3 


GLOSSARY  OF  TERMS  FOR  NETWORK  EXPERIMENTS 


D  -  Link  Demerit  for  TRD 
DE  -  Double-Ended  Time  Transfer 
FMT  -  Frequency  Measurement  Terminal  I 

G  -  Griffiss  Air  Force  Base  Node 
IC  -  Independent  Clock  Timing  Technique 
I/TRD  -  Improved  Time  Reference  Distribution  Timing  Technique 
L[n]  -  Timing  Subsystem  Link  # 

LOCAL  -  Internally-generated  Frequency  Source  (within  TS) 

LOS  -  Line-of-Sight  Microwave  Link 
M/S  -  Master/Slave  Timing  Technique 
MS  -  Mutual  Synchronization  Timing  Technique 
MSTEP  -  Phase  Microstepper 
N  -  Node 

Rb  -  Rudidium  Frequency  Standard 
SE  -  Single-Ended  Time  Transfer 
T  -  Tandem  Network  Configuration 
TRD  -  Time  Reference  Distribution  Timing  Technique 
TRIP  -  Time  Reference  Information  Packet 
TROPO  -  Troposcatter  Link 

V  -  Verona  Node 

VCO  -  Adjustable  Oscillator  Internal  to  Timing  Subsystem 

Y  -  Youngstown  Node 


5-47 


EXPERIMENT  (I).  TWO-NODE  M/S 


MSTEP  MSTEP 


EXPERIMENT  (2):  TWO-NODE  M/S 


5  MHz/Rb  5  MHz /LOCAL 

MSTEP  MSTEP 


EXPERIMENT  (3):  THREE -NODE  TANDEM  M/S 


EXPERIMENT 


(  v  \l 

TROPO 

-Li'  A 

LOS 

V  ■ 

SE 

kLs 

SE 

5  MHz/Rb 

5  MHz/Rb 

5  MHzTlOCAL 

MSTEP 

MSTEP 

MSTEP 

(4) :  THREE 

-NODE  TANDEM  TIME  REFERENCE 

DISTRIBUTION 

R-I 

TROPO 

R-2 

L3/  i/  1LI 

LOS 

R-3 

— <o) 

l  Y  / 

SE 

V  J 

SE 

5  MHz/Rb 

D-5 

5  MHz/Rb 

D-l 

5  MHz/LOCAL 

MSTEP 

MSTEP 

MSTEP 

EXPERIMENT  (5):  THREE -NODE  TANDEM  M/S:  MASTER  WITH  LARGE  FREQUENCY  OFFSET 


5  MHz/Rb  5  MHz/Rb  5  MHz/LOCAL 

MSTEP  MSTEP  MSTEP 

SLEW-2.7994  ns/s 

EXPERIMENT  (6):  THREE -NODE  TANDEM  M/S  WITH  REMOTE  SLAVE  QUARTZ  OSCILLATOR 


5  MHz/Rb  5  MHz/Rb  5  MHz/LOCAL 

MSTEP  MSTEP  VCO 

SLEW-2.7994  ns/s 


Figure  5.20  Summary  of  Network  Configurations 
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as  a  reference  (x.,e.,  zero  frequency  offset),  the  relative  drift 
rates  of  each  of  the  sites  is  plotted.  Small  variations  are 
smoothed  and  incorporated  in  these  "best  fit"  frequency  trend 
lines.  Notice  that  all  rubidium  standards  are  drifting  in  the 
same  direction.  This  frequency  comparison  is  the  basis  for 
evaluating  node  performance  through  measurements  at  any  location. 
This  is  especially  important  since  the  rubidium  standards  are  the 
reference  for  each  site,  and  two  of  the  three  Timing  Subsystems 
(not  GAFB)  use  them  directly  as  an  external  timing  source.  All 
TS  performance  assessment  measurements  are  made  relative  to  the 
site  standards. 

Typically j  an  experiment  evaluation  session  would  proceed 
as  follows: 

•  Select  a  Test  Bed  master  through  which  the 
performance  of  each  node  may  be  compared. 

•  Determine  the  frequency  offset  of  each  site 
frequency  standard  compared  to  the  Test  Bed 
master. 

•  Configure  the  network  for  a  timing  experiment 
including  node  parameters  and  connections  for 
performance  assessment  measurements . 

•  Allow  experiment  to  run  for  at  least  24  hours 
so  that  acquisitional  effects,  and  other  short¬ 
term  perturbations,  do  not  significantly  affect 
the  measurement  series . 

•  Compare  the  time  interval  measurements  of  the 
Timing  Subsystem  adjustable  clock  with  those  of 
the  free-running  node  frequency  standard.  These 
measurements  can  be  collected  through  either  the 
FMT  or  TS,  or  both. 

•  Plot  the  frequency  offset  between  the  TS  adjusted 
clock  and  the  site  standard  (maintain 
directionality) . 

•  Add  or  subtract  the  frequency  offset  of  the  site 
standard  as  compared  to  the  Test  Bed  Master.  The 
result  is  the  total  frequency  offset  of  the  TS 
adjusted  clock  as  compared  to  the  Test  Bed  Master. 
Frequency  comparisons  may  be  either  node-to-node 
or  node-to-Test  Bed  Master. 


This  technique  is  precisely  that  used  to  present  node  evalu¬ 
ation  data  in  Section  5.5.  Table  5-5  summarizes  the  performance 
of  the  network  synchronization  tests.  Notice,  in  directed  con¬ 
trol  experiments,  that  all  nodes  run  at  the  same  total  offset 
compared  to  the  Test  Bed  master,  and  that  all  slave  nodes  run  at 
zero  frequency  offset  compared  to  the  network  master.  This  is 
also  true  for  experiments  5  and  6  where  a  large  frequency  offset 
was  incorporated  into  the  master  node.  Nodal  performance  may  be 
ultimately  compared  to  time  standards  at  NBS,  but  this  exercise 
is  left  to  the  reader.  A  plot  of  LORAN-C  performance,  compared 
to  the  U.S.  National  Time  Standard  (NBS-6) ,  is  shown  in  Figure 
5.22.  This  plot  covers  the  time  period  over  which  the  experi¬ 
ments  were  run  (except  experiment  1).  These  readings,  however, 
are  compared  to  the  Dana,  IN,  LORAN-C  station,  and  interpreting 
time  at  Seneca  would  require  published  phase  delay  tables  and 
additional  calculations.  It  is  not  necessary  to  go  to  this 
amount  of  trouble  unless  the  operator  intended  to  verify  precise 
time  tracking  directly. 

5.6.2  System  Time  Transfer 

The  material  in  the  previous  section  indicates  that,  for 
normal  communication  requirements,  relative  time  synchronization 
of  the  nodes  is  sufficient,  i.e.,  the  node  clocks  need  not  be 
phased  identically  as  long  as  their  mutual  average  frequency 
offsets  are  zero.  On  the  other  hand,  transfer  of  a  time  refer¬ 
ence  throughout  a  network  is  equivalent  to  the  requirement  that 
node  clocks  be  synchronized  with  zero  phase  offset;  and  with  the 
additional  ability  to  remove  ambiguity,  the  aligned  clocks  serve 
as  the  basis  for  a  time-of-day  distribution  system. 

A  brief  review  of  the  parameters  involved  in  system-wide 
time  transfer  will  first  be  given;  more  specific  accounts  of  time 
transfer  methods  are  given  later  in  this  section.  Consider,  in 
particular,  the  transfer  of  time  over  a  single  link  as  depicted 
in  Figure  5.23.  In  this  example,  a  troposcatter  propagation  path 
is  shown  but,  of  course,  other  transmission  media  are  of  direct 
applicability. 

The  primary  function  of  the  link  is  the  transfer  of  digital 
data  between  the  nodes  in  both  directions.  At  any  node,  incoming 
data  is  clocked  into  a  buffer  by  clock  signals  derived  from  the 
receiver-demodulator  bit  tracking  loop.  This  clock  signal 
exhibits  fluctuations  and  drift  behavior  as  a  result  of  transmit 
clock  variations,  medium  variability,  and  tracking  loop  dynamics. 
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Figure  5.22  LORAN-C;  Dana,  IN,  vs.  U.S.  National  Time  Standard  NBS,  Boulder,  CO 


In  a  synchronous  network,  the  received  data  is  later  clocked  out 
of  the  buffer  by  the  node  clock  and  multiplexed  or  switched  with 
data  from  other  terminated  links  for  retransmission;  the  primary 
objective  is  to  coordinate  the  collection  of  node  clocks  so  that 
the  buffers  do  not  overflow  or  deplete.  In  addition,  a  time* 
keeping  function  for  the  node  clock  would  involve  the  desire  to 
meet  the  above  objective  with  zero  phase  offset  between  the  node 
clocks.  Time  is  kept  by  the  clocks  at  each  end  of  the  link  in 
terms  of  a  periodic  sequence  of  time  reference  pulses  (TRP's) 
which  are  synchronous  with  the  high- rate  data  clock.  If  ambi¬ 
guity  issues  are  ignored  in  this  discussion,  it  can  be  simply 
stated  that  the  time  transfer  objective  is  to  align  the  two  TRP 
transmissions.  At  node  B,  this  is  achieved  by  comparing  the 
arrival  time  of  a  pulse  transmitted  from  node  A  with  the  locally- 
generated  pulse  time,  i.e.,  the  node  B  clock  pulse.  Time  trans¬ 
fer  from  A  to  B  can  then  be  implemented  using  this  measurement, 
denoted  tA,  in  one  of  two  ways: 

(1)  Single-ended  transfer,  where  t^  is  viewed  as  a 
combination  of  clock  offset,  average  path/ 
equipment  delay,  plus  a  zero  mean  fluctuating 
component.  That  is,  if  the  average  delay 
through  the  link  is  known  a-priori,  sufficient 
averaging  of  t^  should  yield  the  clock  offset 
and  allow  a  correction  to  be  made. 

(2)  Double-ended  transfers,  where  a  similar  mea¬ 
surement,  denoted  tg ,  is  carried  out  at  node  A,  . 
and  the  quantities  t^  and  tg  are  exchanged  by 
the  two  nodes.  The  difference  parameter  t^-tg, 
computed  at  either  or  both  ends,  is  then  pro¬ 
portional  to  the  clock  offset,  provided  the 
transmission  time  is  identical  in  both  direc¬ 
tions.  To  the  extent  that  this  is  true,  t^-tg 
may  be  used  directly  as  a  clock  correction 
signal. 

In  both  of  these  situations,  the  medium  and  equipment  delay 
variations  control  the  time  transfer  accuracy.  For  single- 
ended  systems,  variability  around  a  long-term  mean,  in  terms  of 
both  magnitude  and  spectral  width,  is  the  essential  ingredient. 
Double-ended  transfer  methods  depend  for  their  success  on  similar 
parameters  expressed  instead  in  terms  of  bidirectional  path  delay 
differences;  namely,  the  time  structure  and  magnitude  of  the  dif¬ 
ference,  along  with  residual  biases  which  are  not  accounted  for 
a-priori . 
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It  should  be  emphasized  that  the  clock  updating  procedure 
is  usually  carried  out  with  quite  long  averages  of  the  measured 
arrival  time  data,  ranging  from  day-long  averages  to  hundreds  of 
seconds  for  quartz  standards.  Thus,  the  short-term  medium  ef¬ 
fects  will  be  virtually  eliminated.  Assume  for  the  moment,  then, 
that  the  clocks  at  either  end  are  aligned  and  much  more  stable 
than  the  incoming  TRP  signal  which  is  perturbed  on  an  instanta¬ 
neous  basis  by  the  medium  fluctuations.  With  this  situation,  the 
path  delay  itself  can  be  established  and,  if  variable,  tracked  In 
time.  In  the  event  of  link  disruptions,  the  demultiplexers  and 
decryption  units  may  lose  sync,  and  the  availability  of  a  network 
time  reference  at  the  node  is  available  to  support  the  reframing 
operation. 

To  further  detail  the  basic  error  computation  chores  for 
each  node  in  a  network,  a  brief  but  concise  account  of  clock 
error  and  path  delay  calculations  for  single-  and  double-ended 
TRIP  exchanges  is  presented.  Refer  to  Section  2.1.1  for  the 
format  of  each  data  package  (TRIP)  as  sent  between  nodes. 

The  ultimate  node  reference  signal  is  the  1  pps  derived  from 
the  on-site  phase-shifted  frequency  standard.  Remember  that  the 
network  time  synchronization  objective  is  the  alignment  of  all 
such  1-pps  references  throughout  the  system.  In  Master/Slave 
and  Mutual  Sync  approaches,  the  relative  variation  of  the  1  pps 
after  buffer  reset  is  the  important  consideration;  that  is,  ab¬ 
solute  phase  or  time  relationships  are  unimportant  throughout  the 
network,  but  accumulated  phase  variations  are  indicative  of 
buffer  fullness.  Therefore,  controlling  one  will  control  the 
other  if  the  station  clock  frequencies  all  originate  from  the 
corrected  station  standard. 

Determining  clock  error  depends  on  the  synchronization  tech¬ 
nique  specified  but,  as  a  general  rule,  it  will  consist  of 
elapsed  time  measurements  from  the  other  end  of  the  link  (from 
a  given  node)  as  well  as  locally-made  time  measurements.  For 
clarity,  we  list  the  time  transfer  parameter  definitions  and 
references.  A  TRIP  timing  exchange  is  shown  in  Figure  5.24. 

All  parameters  are  to  be  determined  as  elapsed  time  measurements. 

t*  =  Node  A  transmitted  TRIP  relative  to  Node  A  reference 
A 

tg  *  Received  Node  B  TRIP  relative  to  Node  A  reference 

t'  =  Node  B  transmitted  TRIP  relative  to  Node  A  reference 

D 

t.  =  Received  Node  A  TRIP  relative  to  Node  B  reference 
A 


TRIP  ARRIVAL 


Figure  5.24  Clock  Error  and  Path  Delay  Calculations  for  Single-  and  Double-Ended 
TRIP  Exchanges 


Figure  5.24  depicts  the  relationships  between  the  TRIP  data 
packets  and  the  actual  elapsed  time  measurements  above.  Single- 
ended  computation  requires  only  a  single  TRIF  transfer  from  the 
"master"  (assume  that  the  receiving  node  is  performing  clock 
error  computation)  to  the  "slave".  For  precise  time  (phase) 
alignment,  we  must  include  a  delay  compensation  term  which  takes 
into  account  all  equipment  and  path  delay  components.  Otherwise, 
phase  errors  will  be  introduced  which  may  accumulate  throughout 
the  network.  The  delay  term  is  not  required  for  frequency 
averaging  techniques . 

Double-ended  computations  require  a  two-way  TRIP  exchange. 
Assuming  that  propagation  delay  is  the  same  in  each  direction, 
clock  error  computations  effectively  eliminate  the  equipment  and 
path  delay  components.  In  fact,  an  alternate  double-ended  com¬ 
putation  may  yield  path  delay  directly. 

Time  Reference  Distribution  involves  an  additional  term  in 
order  to  provide  node-to-node  phase  coherence.  This  term  is  the 
error  of  the  neighboring  node  (or  nodes)  from  which  the  current 
error  computation  is  based  (at  the  local  node) .  Phase  reference 
combining  allows  precise  time  distribution  throughout  a  network 
and  ensures  that  phase  errors  do  not  harmfully  propagate  and/or 
lead  to  potential  network  instability.  If  the  information  over 
multiple  paths  (in  a  large  highly-connected  network)  is  optimally 
combined,  it  is  possible  to  establish  a  greater  timing  accuracy 
at  all  nodes  in  the  network.  Improved  Time  Reference  Distribu¬ 
tion  provides  such  a  capability.  See  Section  5.2.4,  Appendix  A, 
[5.5],  and  [5.6]. 

Substantial  effort  went  into  the  measurement  of  medium 
delay  variability  for  TROPO  and  LOS  links  [5.2].  Although  there 
were  not  too  many  surprises  in  the  resulting  data,  it  should  be 
emphasized  that  many  other  important  issues  were  examined  and 
analyzed  in  the  support  of  these  experiments;  the  critical  evalu¬ 
ation  of  equipment  characteristics  has  brought  forward  some 
particularly  valuable  results,  and,  overall,  the  major  role  of 
the  field  program  has  been  one  of  developing  keenness  of  insight 
into  time  transfer  problems.  Presented  in  this  document  are  the 
many  facets  of  the  time  transfer  and  system  integration  tasks 
that  would  not  have  been  uncovered  without  the  stimulus  of  a 
field  testing  effort. 
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Addressing  more  directly  now  the  results  of  the  mediun 
parameter  tests  are  the  following  conclusions  [5.2] r 


Path  length  variation  for  LOS  links  is  relatively 
unimportant;  according  to  a  rather  modest  data  set, 
it  is  limited  to  about  +10"^  variation  around  the 
nominal  path  length.  The  principal  effect  is  con¬ 
sidered  to  be  refractivity  changes  with  corres¬ 
ponding  variations  in  the  speed  of  propagation.  A 
single -ended  time  transfer  system  with  a  time 
constant  of  a  few  minutes  would  suffice.  Data 
buffering  requirements  amount  to  less  than  1  bit 
over  normal  length  links . 


•  For  TROPO  links,  there  is  considerable  timing  jitter 
in  _he  recovered  clock.  This  has  an  influence  on 
acquisition  performance.  When  the  timing  reference 
is  averaged  over  periods  of  10  minutes  or  more,  the 
long-term  path  length  variations  are  left.  This 
latter  category  exhibits  swings  of  +200  ns  over 
periods  of  several  hours . 


•  Reciprocity  of  the  TROPO  paths  is  not  likely  on  an 
instantaneous  basis.  However,  with  the  short-term 
fluctuations  averaged  out,  forward  and  return  links 
are  expected  to  have  the  same  delay  down  to  the 
20-ns  level.  This  was  confirmed  by  the  results  of 
TROPO  path  length  measurements  in  a  non-diversity 
configuration . 


•  Single-ended  time  transfer  is  not  viable  for  TROPO 
links  unless  long-time  constants  are  used  (e.g., 
several  days)  and  stable  frequency  standards  are 
available  at  both  ends  of  the  link.  With  very  long¬ 
time  constants,  the  tracking  loop  transient  behavior 
becomes  more  severe  in  terms  of  overshoots,  parti¬ 
cularly  when  two  clocks  offset  in  frequency  are 
being  synchronized. 


Finally,  some  comments  on  the  merits  of  different  types 
of  frequency  standards  and  their  performance  when  used  with  TROPO 
and  LOS  links.  Although  time  constants  of  several  hours  or  more 
are  normally  recommended  for  atomic  standards,  there  appears  to 
be  no  particular  advantage  to  such  long  averaging  times  for  LOS 
links.  In  fact,  there  appears  to  be  no  fundamental  argument 
against  the  use  of  quartz  standards  on  most,  or  all,  of  the  LOS 
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links  during  normal  operational  periods.  If  node  clocks  must 
run  in  the  independent  clock  mode,  then  the  long-term  stability 
does  become  important,  but  this  appears  to  be  the  only  real 
factor  against  quartz, standards .  For  TROPO  links,  longer  time 
constants  are  necessary  to  smooth  out  medium  variability.  A  time 
constant  of  an  hour  or  so  is  quite  adequate  for  double-ended 
systems,  and  quartz  clocks  would  be  expected  to  function  satis¬ 
factorily.  However,  for  a  single-ended  TROPO  path,  a  time  con¬ 
stant  of  a  day  or  two  is  necessary;  this,  in  turn,  necessitates 
the  use  of  high-quality  standards  at  the  connected  nodes. 
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SECTION  6 


RECOMMENDATIONS 


This  section  contains  comments  and  suggestions  concerning 
the  Timing  Subsystem  and  related  network  synchronization  topics. 
Included  are  discussions  on  possible  TS  improvements  and  more 
general  future  design  considerations  for  an  advanced  development 
model.  The  topic  of  network  security  and  fallback  modes  of 
operation  are  briefly  reviewed.  Following  is  a  discussion  of 
precise  time  distribution  within  a  synchronous  network.  A  brief 
evaluation  of  the  candidate  timing  techniques  is  revealed  along 
with  implementation  difficulties  or  other  drawbacks  encountered 
during  the  course  of  this  program.  Finally,  topics  for  future 
work  in  network  synchronization  are  presented  including  a  summary 
of  features  considered  desirable  for  a  future  Timing  Subsystem. 

6 . 1  Future  Timing  Subsystem  Design  Considerations 

This  section  summarizes  conclusions  and  recommendations  for 
future  development  of  the  DCS  Timing  Subsystem  based  on  the 
analysis  and  experimentation  carried  out  in  this  program.  Pre¬ 
sented  here  are  the  general  hardware  design  constraints  dictated 
primarily  by  equipment  interfacing  issues.  Included  is  a  brief 
summary  of  the  software  design  and  recommendations  for  further 
development  in  that  area.  Also,  discussions  of  processor  type, 
data  buffer  design  and  placement,  topics  in  error  detection/ 
correction,  and  choice  of  node  frequency  standards  are  presented. 

After  reading  this  section,  it  should  be  clear  that  the 
current  Timing  Subsystem  design  is  capable  of  supporting  a  timing 
function  over  much  larger  networks  than  actually  examined. 

6.1.1  Advanced  Development  Model 

Throughout  this  effort  it  has  been  emphasized  that  the  CNR 
Timing  Subsystem  proved  to  be  versatile,  efficient,  and  flexible 
equipment  easily  configured  and/or  programmed  to  perform  several 
timing  functions.  It  is  felt  that  future  subsystems  should  con¬ 
tinue  to  possess  these  qualities  because  of  the  relatively  low 
cost  oi  n icroprocessor-based  hardware.  In  fact,  it  would  make 
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sense  not  to  sacrifice  a  proven,  dependable  equipment  design 
solely  for  the  purpose  of  streamlining  packaging  or  minimizing 
IC  count.  Instead,  a  significant  portion  of  future  development 
effort  should  be  aimed  at  software  enhancement  through  an  effec¬ 
tive  advanced  software  development  scheme.  It  would  seem  fea¬ 
sible  to  provide  additional  capabilities  in  the  measurement  and 
large-scale  network  performance  platforms,  at  minimum  cost,  while 
retaining  hardware  compatibility  with  present  and  future  equip¬ 
ment  for  simplified  field  efforts. 

Ideally,  many  of  the  important  system-related  decisions 
should  be  made  by  examining  the  behavior  of  a  large-scale  model 
of  the  network.  For  development  purposes,  a  network  with  LOS 
and  TROPO  simulators  may  be  configured  at  one  location,  but  the 
size  of  such  a  network  would  be  limited  by  equipment  availability 
However,  it  should  be  stated  that  more  network- level  testing 
would  be  required  before  any  specific  timing  technique  endorse¬ 
ment  could  be  made  (see  Section  6.4). 

In  summary,  an  advanced  development  model  should  be  enhanced 
over  the  existing  Timing  Subsystem  primarily  in  the  software  end 
of  the  spectrum.  Field  efforts  demonstrated  that  the  subsystem 
hardware  was  capable  of  much  more  demanding  applications  than 
encountered  during  network  tests.  Thus,  the  current  hardware 
design  should  prove  to  be  more  than  adequate  for  future  applica¬ 
tions.  Increased  software  capability  could  then  be  added  at 
minimal  cost  while  improving  the  user  friendliness,  monitora- 
bility,  and  performance  assessment  features  of  the  system. 

6.1.2  Processor  Type 

If  one  were  to  choose  a  single  strength  in  the  Timing  Sub¬ 
system  design,  it  would  be  the  fact  that  the  equipment  was  con¬ 
ceived  around  a  general-purpose  microcomputer  capability;  that 
is,  the  CPU  section  contains  the  elements  essential  to  any  small 
computer  yet  is  ideally  suited  for  the  TS  application.  Moreover, 
flexibility  is  an  integral  part  of  the  design;  the  Timing  Sub¬ 
system  is  fully  programmable  in  nature  and  many  external  events 
are  serviced,  monitored,  or  controlled  by  the  microprocessor. 

The  choice  of  processor  type,  in  general,  is  highly  sub¬ 
jective  in  nature.  This  program  has  sufficiently  demonstrated 
that  a  single  8-bit  "no  frills"  microprocessor  (M6800,  see  [6.1]) 
is  capable  of  handling  the  software  burden  with  adequate  over¬ 
head.  Additionally,  future  software  enhancement  would  not  be 
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limited  by  speed  or  computational  constraints  so  the  current  CPU 
design  would  have  no  trouble  handling  some  added  tasks.  Recall 
that  the  microprocessor  routes  arithmetic  chores  to  an  external 
arithmetic  processor  that  provides  fixed  and  floating-point 
operations.  This  is  an  essential  part  of  the  design  considering 
PLL  implementation,  TRD,  and  other  functions  that  require  a  high¬ 
speed,  32-bit  floating-point  capability  [6.2]. 

The  original  choice  of  microprocessor  type  was  strongly 
influenced  by  existing  development  capability  at  RADC  as  well 
as  more  general  compatibility  issues.  Whichever  microprocessor 
were  to  be  used,  only  minor  differences  in  CPU  design  and  inter¬ 
rupt  protocol,  for  example,  would  surface.  Other  differences 
would  make  themselves  evident  in  software,  but  overall  design  and 
development  costs  for  any  of  the  more  popular  8-bit  micros  would 
be  comparable. 

As  an  alternate  view  of  the  situation,  one  may  argue  that 
one  of  the  newer,  more  powerful  16 -bit  microprocessor  devices 
would  be  more  suitable  in  the  Timing  Subsystem  design.  However, 
it  is  clear  that  the  8-bit  microprocessor  is  not  taxed  beyond  its 
capability;  in  fact,  sufficient  headroom  exists  to  allow  advanced 
development  without  the  need  for  redesigning  the  computer  section 
of  the  Timing  Subsystem.  Generally,  16 -bit  micros  provides  10  - 
25  times  more  throughput  than  most  8-bit  devices,  but  it  is 
certain  that  much  of  this  speed  and  capability  would  go  unused. 

In  summary,  almost  any  popular  8-bit  microprocessor  would 
do  the  job  satisfactorily  -  providing  a  complete  family  of  I/O 
support  devices  were  available.  The  current  Timing  Subsystem 
design  and  processing  load  is  fully  within  the  limitations  of 
the  Motorola  6800  device  family  with  ample  overhead,  even  in 
worst-case  conditions.  For  a  look  at  the  typical  processing 
times  per  one-second  iteration,  see  Table  6-1.  This  table 
serves  to  illustrate  and  partition  the  execution  times  of  var¬ 
ious  program  sections  encountered  each  second.  Recall  that 
Timing  Subsystem  operation  is  based  on  a  once-per-second  repeti¬ 
tion  rate  and  that  several  of  the  basic  program  modules  are  syn¬ 
chronous  to  that  rate.  Other  asynchronous  or  conditional  events 
are  noted  in  the  comments  column  and  should  be  considered  when 
computing  the  total-per-second  execution  time  based  on  a  parti¬ 
cular  preprogrammed  TS  configuration.  Notice,  however,  that 
for  a  typical  processing  load  encountered  during  a  Master/Slave 
experiment  terminating  two  links,  the  total  execution  time  is 
approximately  36.45  ms  occupied  per  second.  The  largest 
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TABLE  6-1 


TYPICAL  PER-SECOND  EXECUTION  TIMES  FOR 
TIMING  SUBSYSTEM  MICROPROCESSOR* 


Task 

Execution 

Time 

Comments 

Interrupt  Dispatcher 
Overhead  and 

Interrupt  Latency 

180  ps 

For  a  typical  configuration 

Once -per -second  Service 
Performance  Assessment 
Service 

Fault  Check 

12,500  ps 

Checks  for  buffer  ovf,  unf, 
reference  change,  etc. 

Keeps  running  averages  of 
clock  error  and  error  com¬ 
pared  to  3  AUXTRP's 

Begins  link  1  transmit 
sequence 

UDdates  clock  correction 

Transmit  Bytes 

40  jis/byte 

2,680  ps  for  67  bytes  total 
per  link 

16,080  ps  for  6  link  max 

Receive  Bytes 

50  ps/byte 

3,350  ps  for  67  bytes  total 
per  link 

20,100  ps  for  6  link  max 

Frame  Processing 

55  pa/link 

Frame  assembly  overhead  per 
link 

Reference  Selection 

PLL  Loop  Update 

■ 

Completes  clock  error  for 

SE  or  DE  transfers 
Determines  operating  mode 
Implements  PLL  algorithm 
Performs  timing  technique 
algorithm 

Assembles  outgoing  TRIP'S 

"  - - 

Mlsc.  Bookkeeping 

120  ps/link 

Per  operating  link 

IND  CLK 

Timing  M/S 

Algorithm  MS 

(Includes 

Error 

Computation)  Improved 

TRE 

20  us 

1  Mostly  bookkeeping 

1.100  pS 

520  ps 

Finds  best  path  to  master, 
determines  hierarchy 

3,275  ps 

Additional  computation  and 
bookkeeping  overhead 

Total  (per  second) 

36,450  ps 

For  a  typical  2- link 
configuration  with  M/S 

64,485 

Worst  Case  (6  links  Im¬ 
proved  TRD) 

i 


Motorola  M6800  with  1-MHz  clock. 
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computational  burden  can  be  imposed  by  running  Improved  TRD 
while  terminating  six  active  links.  The  resulting  execution 
time  is  approximately  64.49  ms  per  second.  Field  test  configu¬ 
rations  resulted  in  processing  times  similar  to  the  former 
example.  Execution  times  are  determined  by  adding  instruction 
execution  times  or,  in  some  cases,  by  direct  measurement. 

6.1.3  Data  Buffer  Design  and  Placement 

Now  look  at  the  buffering  capability  provided  by  the  Timing 
Subsystem  prototype.  The  buffers  in  question  are  used  primarily 
for  protection  against  timing  transients  which  may  occur  in  a 
system  with  coordinated  clocks  or,  more  likely,  to  take  up  the 
differences  in  data  transmitted  between  nodes  when  independent 
clocks  are  set  up  in  the  network. 

A  few  general  comments  will  help  to  orient  the  reader  before 
more  specific  recommendations  are  put  forward  [6.3]: 

•  Present  technology  allows  buffers  for  level  2  (up 
to  13  Mb/s)  to  be  built  at  only  a  marginally  higher 
cost  than  buffers  for  level  1  or  below.  Above  about 
IK  bits  of  storage  there  is  virtually  no  cost  differ¬ 
ential  and,  more  important,  the  incremental  cost  of 
adding  storage  beyond  this  value  is  quite  small  com¬ 
pared  with  the  initial  cost  of  control  circuitry. 

•  When  buffering  is  carried  out  at  one  of  the  lower 
levels,  all  upstream  equipment  must  be  clocked  from 
the  recovered  modem  clock  which  will  exhibit  jitter 
and  other  medium  effects  (e.g.,  frequency  offset  for 
satellites).  To  a- -oid  the  presence  of  this  jitter 
in  equipment,  it  must  be  placed  on  the  downstream 
side  of  the  buffer. 

•  At  some  stations,  switching  and  combining  of  data 
streams  from  several  remote  sources  must  be  achieved. 

This  can  only  be  done  with  synchronously  clocked  data, 
implying  that  the  data  must  have  already  passed 
through  a  buffer  by  the  time  it  is  presented  for 
combining.  For  example,  if  digroups  (1.544  Mb/s) 
with  different  geographical  origins  are  to  be  com¬ 
bined  at  a  level-2  MUX,  they  must  have  been  sub¬ 
jected  to  buffering  either  at  level  2  or  at  level  1. 
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•  AC  Che  ouCpuC  of  Che  radio,  daCa  musC  be  buffered 

in  such  a  way  ChaC  resefs  correspond  Co  Che  deleCion 
or  inserCion  of  an  inCeger  number  of  level-1  and 
level-2  frames.  We  dismiss  Che  possibiliCy  of  ex- 
Cending  Chis  concepC  Co  lower  level  submulciplexers 
because  of  Che  increased  sCorage  requiremencs .  How¬ 
ever,  Che  possible  avoidance  of  level-l/level-2  TDM 
resynchronizaCion  is  in  icself  appealing  enough, 
parCicularly  when  a  major  porCion  of  Che  Craffic  is 
PCM  voice.  In  Cerms  of  Che  effecCs  on  voice  channels, 
iC  was  our  objecCive  Co  reduce  Che  reseCCing  operaCion 
Co  Che  loss  of,  aC  mosC,  Cwo  PCM  samples  for  each 
voice  channel  passing  Chrough  Che  buffer. 

•  To  successfully  employ  such  a  sCraCegy,  iC  is  neces¬ 
sary  Co  know  Che  TDM  frame  formaC,  and  Co  examine  Che 
consequences  of  implemencing  mulCiple  frame  reseCs 
wich  buffers  aC  different  levels,  e.g.,  aC  level  2. 

A  jump  by  a  whole  level-2  TDM  frame  may  noC  Chrow  ChaC 
DEMUX  ouC-of-sync,  buC  sCill  noC  safeguard  Che  level-1 
DEMUX's.  AlCernaCively,  if  Che  buffers  are  aC  level-1, 
such  reseCs  do  noC  affecC  Che  level-2  DEMUX's  in  any 
way. 

•  The  level-2  TDM  race  specificaCions  allow  for  inpuC 
porC  scrapping  (i.e.,  ouCpuC  ports  at  the  DEMUX); 
hence,  the  buffer  design  must  accommodate  various 
rates  if  it  is  to  be  used  at  level  1.  This  is  rather 
awkward  when  one  considers  that  the  total  number  of 
bits  of  storage  required  at  a  level-2  DEMUX  output  is 
invariant  regardless  of  the  division  of  the  total  MBS 
bit  rate  into  various  port  rates,  strapped  or  other¬ 
wise.  Obviously,  the  designer  is  forced  to  develop 
an  8-port  flexible  buffer  unit  with  programmable  rate 
structure  (and  movable  storage  boundaries),  or  he 
must  design  for  the  worst-case  rate  (i.e.,  maximum) 
and  develop  other  identical  buffers  to  satisfy  the 
worst-case  number  of  ports.  These  design  issues  give 
added  weight  to  the  selection  of  a  level-2  buffer 
design. 

•  Other  considerations  mainly  involve  cost  and  reli¬ 
ability  of  a  few  large  fast  buffers  vs.  a  larger 
number  of  smaller  slower  buffers  of  comparable  total 
capacity.  A  large  buffer  is  probably  cheaper  than  a 
number  of  smaller  ones  of  similar  total  capacity, 
unless  reliability  requirements  (MTBF  is  probably  a 
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function  of  size)  or  the  high  data  rate  required 
dictate  a  more  expensive  approach  (either  a  dif¬ 
ferent  technology  or  more  expensive  components) . 

The  consequences  of  a  failure  of  a  larger  buffer 
are  more  serious  than  those  of  a  failure  in  one  of 
a  number  of  smaller  buffers  which  might  be  used 
instead . 

There  exist  six  elastic  stores  in  each  Timing  Subsystem 
corresponding  to  one  for  each  of  the  terminated  links  at  a  node. 
These  buffers  are  placed  at  the  output  of  the  digital  radio  de¬ 
multiplexer  and,  therefore,  must  operate  at  speeds  up  to  13  Mb/s. 
Strictly  speaking,  the  DCS  specifications  implicitly  require  two 
such  buffers  at  each  link  termination,  since  the  DCS  radios  are 
designed  to  accept  two  separate  MBS  data  lines.  However,  only 
single  line  terminations  were  implemented  at  the  RADC  Test  Bed 
and,  as  an  experimental  model,  the  total  subsystem  buffer  capa¬ 
city  was  more  useful  when  structured  as  6  double-length  buffers 
rather  than  12  single-length  buffers. 

The  buffer  size  (2K  bits)  was  determined  on  the  basis  of  a 
minimum  size  requirement  for  independent  clock  operation. 

Starting  half  full,  with  two  parts  in  10^^-  difference  in  input 
and  output  clock  frequencies,  it  will  overflow  or  underflow  in 
1100  hours  (about  46  days);  with  one  part  in  10^  relative  clock 
frequency,  the  time  is  2.2  hours. 

An  attractive  possible  application  for  relatively  large 
buffers  is  the  preservation  of  multiplexer  synchronization  during 
buffer  reset.  If  buffer  capacity  exceeds  the  length  of  the 
framing  pattern  (e.g.,  superframe  or  super- super frame) ,  when  it 
gets  nearly  full,  its  fullness  can  be  decreased  in  one  jump  by 
a  number  of  bits  equal  to  the  framing  pattern  length.  CNR  has 
examined  a  buffer  design  to  include  this  feature.  If  the  TDM 
1193  superframe  needs  to  be  preserved,  the  total  buffer  size  must 
be  in  excess  of  25K  bits  and  the  step  resets  must  be  of  a  corres¬ 
ponding  magnitude.  It  would  appear,  however,  that  the  1193  super¬ 
frame  carries  only  pulse  stuffing  information,  which  will  not  be 
relevant  in  a  synchronous  network.  Obviously,  it  is  necessary  to 
have  additional  information  concerning  the  1193  frame  format  to 
pursue  the  design  further  [6.5]. 

A  buffer  design  (Figure  6.1),  using  cascaded  64x4  FIFO 
modules,  is  used.  Using  FIFO's  rated  to  operate  at  data  rates 
up  to  10?  words  per  second,  or  40  Mb/s,  this  buffer  can  easily 
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handle  the  12.928-Mb/s  data  rate  of  the  level-2  multiplexer  MBS. 
Four-bit  serial-to-parallel  and  parallel-to-serial  registers 
interface  the  input  and  output  bit  streams  with  the  4-bit  wide 
FIFO,  which  have  TTL- compatible  inputs  and  outputs.  A  numeric 
display  indicates  buffer  fullness  in  eighths  of  full  capacity: 

1  *  less  than  1/8  full;  2  =  1/8  to  1/4,  etc.;  8  ■  more  than  7/8  full. 
A  reset  circuit  which  initially  loads  the  FIFO  half-full  is  in¬ 
cluded;  the  output  clock  is  gated  off  and  the  FIFO  cleared; 
when  the  FIFO  is  half-full,  the  output  clock  is  allowed  to  start 
[6.4]. 

This  design  proved  to  be  quite  reliable  in  the  field. 

Buffer  fullness  status  was  available  to  the  computer  and,  sub¬ 
sequently,  to  the  front  display  panel  and  the  operator. 

Continued  use  of  this  design  is  recommended. 

6.1.4  Error  Detection/Correction 

The  transfer  of  clock  data  throughout  the  network  requires 
some  thought,  particularly  with  regard  to  error  control  and  cen¬ 
tralized  network  control. 

Reliability  for  the  clock  control  data  link  is  paramount. 

It  is  quite  obvious  that  the  data  being  transferred  between  nodes 
is  highly  intolerant  of  errors;  the  adaptive  reorganization  merit 
and  ranking  parameters,  for  example,  must  be  carefully  protected 
to  eliminate  the  possibility  of  inadvertent  network  timing  errors 
or  reconfiguration. 

The  task  of  designing  the  data  link  is  complicated  even 
further  by  the  following  considerations: 

•  Error  patterns  observed  on  the  links  will  tend  to  be 
burst-like  as  a  result  of  channel  fading,  particularlv 
for  TROPO  and  LOS  systems. 

•  Although  the  DCS  specifications  call  for  quite  low 
error  rates  (e.g.,  5x10"”  for  LOS,  5  x  10~5  for 
TROPO,  99.997o  of  the  time)  [6.5],  data  link  integrity 
is  even  more  important  during  periods  of  jamming  when 
the  error  rates  will  be  much  higher.  Therefore,  a 
high  tolerance  to  errors  is  necessary. 


A 


6-9 


From  these  statements  one  can  formulate  some  important 
guidelines  which  apply  to  our  data  link  design.  Briefly,  there 
are  certain  data  items  that  must  be  delivered  with  virtually 
zero  error  probability  or,  as  an  alternative,  not  at  all;  i.e., 
gaps  in  the  clock  updating  procedure  are  preferable  to  the  use 
of  invalid  data.  Then  it  must  be  accepted  that,  during  times  of 
stress,  the  raw  bit  error  rate  may  be  quite  high  (e.g.,  approach¬ 
ing  0.5).  These  two  facets  of  the  data  communications  design 
task  are  rather  ominous  considering  the  narrow  range  of  options 
they  leave  open. 

The  necessity  of  achieving  close  to  zero  error  probability 
strongly  suggests  that  a  dependable  error  detection  scheme  is 
mandatory;  error  correction  is  of  value  since  it  may  be  useful 
in  bolstering  the  effective  data  transfer  rate.  The  use  of 
several  data  modes  is  also  advocated.  One  can  exploit  the  fact 
that,  in  a  heavy  jamming  situation,  only  the  more  essential  data 
need  be  conveyed  between  nodes.  Thus,  one  should  contemplate  a 
backup  low  data  rate  mode  of  operation.  The  unusea  capacity 
would  then  serve  to  enhance  the  reliability  of  the  remaining 
data  bits  by  allowing  additional  redundancy. 

Error  will  occur  in  the  transmission  of  data  from  one  point 
to  another  and,  if  the  information  content  is  highly  critical  as 
it  is  in  the  case  of  clock  control,  plans  must  be  made  for  deal¬ 
ing  with  the  errors .  There  are  two  fundamental  categories  for 
improving  the  reliability  of  a  data  link.  The  first,  commonly 
called  ARQ  (Automatic  Repeat  Request) ,  involves  transmission  by 
blocks.  Each  block  has  a  number  of  parity  bits  appended.  At  the 
receiving  end,  the  parity  bits  are  recomputed  from  the  data  and 
compared  with  the  received  parity  bits .  If  there  are  no  dis¬ 
crepancies,  the  block  is  accepted;  otherwise,  the  sending  station 
is  notified  and  the  complete  block  must  be  transmitted.  This  is 
referred  to  as  an  error  detection  scheme.  It  is  the  job  of  the 
decoder  to  detect  the  presence  of  most  errors,  while  correction 
may  be  carried  out  by  retransmission. 

Forward  Error  Control  (FEC) ,  on  the  other  hand,  is  more 
complicated  and  involves  attempts  by  the  decoder  to  determine  the 
location  of  the  errors  from  the  pattern  of  discrepancies  between 
received  and  recalculated  parity  bits.  With  FEC  there  is  an 
error  correction  mechanism.  Depending  on  the  nature  of  the  error 
patterns,  FEC  systems  may  be  able  to  correct  a  maximum  number 
(say,  t)  random  errors  or,  alternatively,  a  burst  of  errors 
having  a  span  of  B  bits.  Composite  error  detection/error 
correction  schemes  using  both  ARQ  and  FEC  can  also  be  devised. 
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In  most  situations,  particularly  TROPO,  the  errors  will  not 
be  independent  random  events,  primarily  because  of  channel  fades 
and  we  must  evaluate  and  compare  the  performance  of  ARQ  and  FEC 
techniques  carefully  to  see  how  they  perform  in  a  burst  error 
environment . 

While  ARQ  errors  may  be  corrected  by  retransmission  of  a 
block,  the  FEC  must  attempt  correction  on  the  fly.  Thus  we 
see  that,  although  FEC  may  not  be  able  to  achieve  such  low 
overall  error  rates  as  ARQ,  the  latter  has  a  variable  and 
possibly  low  transmission  efficiency.  There  is  no  doubt  that 
a  combination  of  both  has  merit  [6.6] ,[6.7] . 

There  are  several  ARQ  variants.  By  far,  the  most  common  is 
the  stop  and  wait  system,  whereby  the  transmitter  waits  after 
each  block  for  a  positive  acknowledgement  before  sending  the 
next  block.  Then  there  are  two  versions  of  the  so-called  con¬ 
tinuous  ARQ  approach.  With  sequential  numbering  of  the  outgoing 
blocks,  the  receiver  is  able  to  request  retransmission  of  a 
particular  block.  This  can  be  done  in  a  pull-back  manner;  the 
transmitter  discounts  any  higher  numbered  blocks  it  may  have 
sent  in  the  interim,  and  retransmits  a  complete  sequence  of 
blocks,  starting  at  the  block  found  to  be  4-n  error.  This  keeps 
the  blocks  in  sequence.  Alternatively,  the  transmitter  can 
retransmit  only  the  block  found  to  be  in  error,  in  which  case 
the  blocks  would  be  received  out  of  order. 

FEC  performance  is  more  sensitive  to  the  distributional 
nature  of  the  errors,  e.g.,  burst  or  random.  The  easiest  way 
of  achieving  protection  against  both  kinds  is  to  use  interleaved 
codes;  with  a  relatively  low  cost  approach,  modest  improvements 
can  be  achieved  with  this  method.  The  FEC  schemes  available  are 
successful  to  one  degree  or  another  in  reducing  the  bit  error 
rate.  But  the  fact  remains  that  errors  must  be  virtually  eli¬ 
minated  for  some  of  the  more  sensitive  clock  data.  Consequently, 
a  high  level  of  error  detection  performance  is  still  required. 

The  preceding  paragraph  suggests  that  the  principle  of 
operation  for  the  data  link  should  be  based  on  ARQ  with  a  very 
low  residual  undetected  error  rate  by  judicious  choice  of  parity 
check  code.  However,  it  is  apparent  that  additional  FEC  capa¬ 
bilities  will  be  required  between  the  SDLC  line  controller  and 
the  service  channel  multiplexer  to  maintain  good  performance 
during  periods  of  jamming  when  the  error  rate  will  be  quite 
high. 
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A  full  investigation  of  the  requirements  for  such  an  EEC 
unit  appears  necessary.  However,  three  key  topics  can  be  itemi¬ 
zed  at  this  point: 

•  Delay  through  the  FEC  must  be  carefully  controlled 
if  the  clock  data  channel  is  being  used  to  convey 
node  time,  as  in  option  B  described  in  [6.5]. 

•  With  the  channel  data  rate  selected  as  4  kb/s,  and 
an  information  rate  that  is  on  the  order  of  500  b/s, 
there  is  significant  redundant  channel  capacity 
available  for  use  by  the  coding  scheme. 

•  Contingency  plans  can  be  formulated  for  minimizing 
the  data  transfer  requirements  during  jamming  situ¬ 
ations.  Such  a  reduction  will  allow  the  use  of 
even  lower  rate  codes  in  the  FEC  application. 

Finally,  note  that  the  burst  nature  of  the  error  distribu¬ 
tions  existing  on  TROPO  links  creates  additional  difficulties 
which  can  only  be  solved  with  specialized  coding  and  decoding 
equipment . 

6.1.5  Node  Frequency  Standards 

This  section  is  intended  to  clarify  the  difference  between 
a  disciplined  oscillator  used  in  a  network  timing  experiment  and 
a  frequency  standard,  including  when  each  should  be  used. 

A  frequency  source  option  was  used  on  one  of  the  Timing  Sub¬ 
systems.  It  consisted  of  a  Voltage-Controlled  Crystal  Oscillator 
(VCO)  with  a  D/A  interface  to  provide  computer  control  of  this 
device.  The  reason  for  this  option  was  twofold.  First,  it  was 
to  be  determined  by  experimentation  if  a  disciplined  quartz 
oscillator  could  perform  well  when  slaved  to  a  more  elaborate 
frequency  standard.  Second,  if  such  a  frequency  source  was 
cost-effective  in  a  large  network,  would  it  be  feasible  to  in¬ 
corporate  VCO's  at  many  less  critical  nodes;  that  is,  nodes  that 
are  not  directly  contributing  to  the  timing  function,  without 
seriously  compromising  the  synchronization  performance  of  the 
network? 

Although  each  test  site  had  on-hand  a  rubidium  frequency 
standard  at  the  minimum,  for  the  VCO  performance  test,  the  fre¬ 
quency  standards  served  only  as  a  reference  for  clock  comparison; 
i.e.,  the  Timing  Subsystem  that  had  its  own  internal  oscillator 
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did  not  need  an  external  frequency  source  applied  for  the  clock 
adjustment  mechanism.  Instead,  this  VCO  operated  at  the  same 
center  frequency  (5  MHz)  and  had  an  adjustment  range  of 
1.6  parts  x 10?  (similar  to  that  of  the  phase  microstepper). 

A  small  penalty  in  software  overhead  to  develop  a  driver  was 
encountered  over  that  for  the  phase  microstepper. 

Summarizing  the  findings,  it  was  noticed  that  the  VCO,  when 
operated  as  a  slave  (i.e.,  disciplined  mode),  performed  well 
and  certainly  within  the  limits  for  overall  network  accuracy 
as  specified  for  the  DCS.  Also,  considering  the  cost  to  install 
rubidium  frequency  standards  (about  five  times  higher  than  a 
VCO)  and  phase  microsteppers  at  each  node,  it  would  be  good 
practice  to  consider  the  use  of  quartz  VCO's  at  many  of  the  less 
important  nodes.  However,  it  is  strongly  recommended  that  the 
higher  quality  frequency  sources  be  distributed  in  some  logical 
fashion. 

6.1.6  Software  Design 

Timing  Subsystem  software  is  highly  modular  in  design  due 
largely  to  the  asynchronism  of  events  external  to  the  processor. 
Timing  is  based  on  a  once-per-second  update  rate  provided  by 
dividing  down  the  adjusted  5-MHz  node  frequency  source.  Incoming 
and  outgoing  TRIP'S  interrupt  the  processor  for  service  when  two 
bytes  are  buffered  and  ready  to  be  transferred.  Prioritized 
events  are  handled  by  an  Interrupt  Dispatcher  which  is  in  charge 
of  determining  the  sequence  of  module  execution  according  to  a 
set  of  predetermined  priorities.  A  small  system  monitor,  em¬ 
bedded  in  the  software,  provides  basic  interrogation  and  debug¬ 
ging  capabilities  as  well  as  being  the  interface  between  the 
operator  and  the  Timing  Subsystem  [6.8]. 

The  success  of  the  software  effort  is  due  largely  to  the 
modularity  of  the  package.  Routines  are  divided  along  function¬ 
ality  lines,  which  simplified  software  planning  and  development. 
All  code  is  reentrant. 

It  is  strongly  recommended  that  future  work  allow  the 
enhancement  of  existing  programs  to  avoid  a  completely  new 
hardware  and  software  design.  For  example,  an  advanced  develop¬ 
ment  model  may  include  the  following  salient  features: 
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•  An  enhanced  operating  system  to  increase  user 
friendliness  and  provide  English- like  commands 
and  more  natural  dialog. 

•  Increased  tolerance  to  perturbed  incoming  data 
streams.  (This  implies  error  correction  facilities.) 

•  More  complete  performance  assessment  capabilities. 

Since  this  is  the  only  means  by  which  long-term 
performance  is  verified,  it  makes  sense  to  provide 
a  comprehensive  evaluation  of  nodal  clock  behavior 
for  a  given  length  of  time.  External  clock  compari¬ 
sons  should  also  be  maintained  in  this  fashion. 

•  A  simple  efficient  network  management  scheme  would 
be  most  valuable  in  larger  networks.  For  this  pro¬ 
gram,  node  management  was  carried  out  on  a  node-by¬ 
node  basis  by  telephone  connection  through  the 
Frequency  Measurement  Terminals  located  at  each  site. 
However,  a  management  protocol  could  be  devised  to 
set  parameters  remotely  via  bit  patterns  contained 

in  unused  portions  of  the  TRIP  format  or  by  extending 
the  TRIP  itself.  (There  exists  sufficient  overhead 
for  up  to  a  tenfold  increase  in  TRIP  length.) 

These  improvements  could  be  built  around  the  already  solid 
and  field-proven  processing  base.  Interrupt-driven  routines  are 
scheduled  by  a  software  Interrupt  Dispatcher,  and  the  system 
monitor  responds  to  the  operator  when  scheduled  processes  have 
completed.  (See  Section  2.2  of  this  report  and  Section  3  of 
[6.8]  for  a  more  complete  analysis  of  the  current  software 
design.)  Some  of  the  more  fundamental  routines  which  are  cur¬ 
rently  employed  and  would  remain  as  the  foundation  of  an  advanced 
model  are: 

•  Link  I/O  byte-oriented  operations.  These  are  interrupt- 
driven  routines  for  both  transmit  and  receive  sequences. 
Programs  are  based  on  a  single  chip  implementation  of 
the  SDLC  protocol  (including  16-bit  CRC  check  for 
error  detection) . 

•  Clock  error  computation  based  on  single-ended  or 
double-ended  transfers.  Additional  computation 
(double-ended)  produces  path  delay  or  TRD  error 
information. 
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•  Normalization  for  computed  clock  errors. 

•  Software  delay  routine.  This  calibrated  multi-stage 
loop  coarse  adjusts  the  node  clock  with  an  accuracy 
of  +4fiS  over  a  one-second  range. 

•  PLL  algorithm.  Includes  PLL  state  equations  and 
programmable  time  and  gain  constants. 

•  Interrupt  Dispatcher.  Schedules  routine  execution 
based  on  a  predetermined  priority  scheme.  Relays 
control  to  the  operating  system  when  CPU  is  idle. 

(See  Section  3.2.1  of  [6.8].) 

Summarizing,  it  is  encouraged  that  an  advanced  development 
model  incorporate  several  modifications  to  the  original  software 
design.  These  changes  would  not  affect  the  underlying  processing 
routines  but  serve  to  enhance  an  already  efficient  and  reliable 
software  package. 

6.2  Network  Survivability 

The  DCS  network  must  be  designed  for  satisfactory  operation 
under  conditions  of  hostility.  There  is  an  underlying  premise, 
therefore,  that  no  single  node  should  be  critical  to  the  func¬ 
tioning  of  the  remainder  of  the  network.  When  one  or  more  nodes 
are  eliminated,  the  surviving  nodes  should  be  capable  of  con¬ 
tinued  operation. 

In  Section  6.2.1  we  examine  the  issues  of  survivability  and 
security  by  comparing  the  different  clock  control  techniques  in 
stress  situations  requiring  reorganization  of  the  timing  distri¬ 
bution  scheme.  In  Section  6.2.2  we  cover  the  subject  of  backup 
modes,  where  it  is  envisioned  that  normal  operation  with,  say. 
Master /Slave,  Mutual  Sync,  or  TRD,  is  ruled  out  for  the  complete 
network,  or  at  least  for  large  portions  of  it. 

6.2.1  Self-Organizing  Features 

When  segments  of  the  network  are  in  a  condition  of  stress, 
it  is  likely  that  the  timing  distribution  tree  for  the  remainder 
of  the  network  will  have  to  be  reorganized.  This  is  the  essence 
of  survivability,  and  its  importance  for  a  military  network  is 
patently  clear.  Of  course,  there  are  more  mundane  reasons  for 
implementing  automatic  time  distribution  recovery;  clock  and 
transmission  equipment  failures  in  the  network  are  inevitable 
and  contingency  plans  must  be  formulated  to  cover  these  events. 
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The  purpose  of  this  section  is  to  assess  the  relative 
degree  to  which  the  synchronization  technique  provides  a  sur- 
vivable  network.  Comparison  of  directed  and  mutual  control  can 
be  made  in  cases  where  references  are  not  available.  For  the 
references  which  are  available,  mutual  and  directed  control 
techniques  differ  considerably  because  each  node  in  the  mutual 
system  is  always  using  all  of  its  incoming  link  data,  whereas  a 
node  in  the  directed  control  system  derives  timing  from  only  one 
of  the  connected  links.  Transients  can  occur  in  the  mutual 
system,  but  the  directed  control  system  is  affected  only  if  the 
reference  on  the  chosen  timing  link  is  removed.  With  either 
technique,  those  features  which  tend  to  minimize  instantaneous 
frequency  and  phase  errors  will  tend  to  improve  network  surviva¬ 
bility  [6.9] , [6 .10] .  With  directed  control,  the  sooner  a  node 
can  find  a  new  reference  once  an  old  one  has  been  removed,  the 
better  from  a  survivability  standpoint.  Thus,  self-organization 
is  quite  important  to  the  survivability  of  a  direct  control 
network. 

Both  mutual  and  directed  control  systems  have  similar  prob¬ 
lems  when  no  references  are  available.  Both  must  rely  on 
previous  clock  correction  history  in  order  to  maintain  synchro¬ 
nism.  Accordingly,  the  stability  of  the  local  clocks  is  of 
critical  importance.  However,  performance  with  no  references 
available  is  based  on  the  history  of  external  stress  events  and 
individual  clock  behavior.  Therefore,  those  features  which 
reduce  the  complexity  of  this  error  history  during  normal  opera¬ 
tion  will  increase  the  potential  for  survivability  during  periods 
when  references  are  not  available  [6.9]. 

Of  the  synchronization  techniques  considered,  the  TRD 
approach  is  unique  in  the  survivability  features  it  offers.  It 
is  sufficient  to  say  here  that  the  capacity  to  automatically 
reorganize  is  the  feature  of  TRD  that  sets  it  apart  from  the 
others;  namely,  Master/Slave,  Mutual  Sync,  and  Independent 
Clocks.  This  self-organizing  capability  should  be  recognized 
as  a  characteristic  that  does  not  necessarily  have  to  be  asso¬ 
ciated  with  TRD;  that  is,  the  time  distribution  functions  and 
the  reorganization  functions  are  essentially  separate.  We  can 
contemplate  the  development  of  a  relative  time  synchronization 
scheme  which  has  these  self -structuring  features;  it  would  be 
equivalent  then  to  a  Mas ter /Slave  scheme  with  an  adaptively 
reorganized  distribution  tree. 
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It  can  be  safely  concluded  that  the  self -organizing  aspects 
of  the  currently  formulated  TRD  technique  are  of  great  benefit, 
and  bring  to  the  overall  network  a  level  of  resilience  that 
cannot  be  matched  by  other  approaches.  Of  the  remaining  choices, 
the  Independent  Clock  method  is  in  a  class  of  its  own  and  need 
not  be  discussed.  Then,  the  Mutual  Synchronization  technique 
should  be  considered  as  having  reasonable  survivability  charac¬ 
teristics.  Despite  the  potential  for  operation  without  the  need 
for  a  clock  control  data  link,  there  are  questions  of  stability 
which  arise  when  the  structure  of  the  network  is  undergoing 
change.  These  issues  are  best  addressed  with  large-scale  network 
simulation  tools;  it  is  not  likely  that  the  small  network,  set  up 
as  part  of  this  contract,  will  offer  insight  to  behavior  for  net¬ 
work  problems  such  as  these.  Finally,  it  is  recognized  that  the 
Master/Slave  technique,  at  least  in  its  classical  form,  is  not 
well-suited  to  coping  with  network  stress. 

There  is  a  temptation  to  treat  these  candidate  techniques  in 
isolation,  as  separate  and  unalterable  methodologies.  But,  as 
pointed  out  above,  certain  features  of  one  may  be  applicable  to 
others.  The  self -organizing  features  of  TRD  should  not  be  asso¬ 
ciated  with  TRD  exclusively,  for  example.  Another  important 
point  is  that  reorganization  without  control  data  linking  is  a 
very  desirable  goal;  that  is,  it  may  be  feasible  to  formulate 
reference  selection  rules  on  the  basis  of  the  received  TRP  sig¬ 
nals  alone.  Clearly,  there  will  be  a  sacrifice  in  terms  of 
selection  optimality.  On  the  other  hand,  the  physical  structure 
of  the  DCS  network  is  likely  to  remain  relatively  invariant,  and 
preplanned  timing  recovery  strategies,  including  reference  selec¬ 
tion,  should  be  easy  to  test  by  means  of  simulation.  Furthermore, 
with  a  reliable  enough  backup  mode  of  operation,  the  reference 
selection  process  could  be  undertaken  at  a  fairly  leisurely  pace 
to  minimize  the  possibility  of  network  instability. 

6.2.2  Backup  Modes  of  Operation 

When  one  considers  the  topic  of  survivability,  the  idea  of 
a  backup  mode  is  immediately  brought  to  mind.  Here  we  will  be 
concentrating  on  two  particular  backup  configurations.  It  should 
not  be  concluded  that  these  are  the  only  two,  but  they  are 
generally  considered  as  likely  candidates.  These  backup  concepts 
involve: 


•  Use  of  AN/GSQ-183  timing  terminal;  i.e.,  a 
LORAN-C  based  external  frequency  source 
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•  Network  operation  with  highly  stable  independent 
standards 

Both  modes  will  be  considered  subsequently . 

First  is  an  outline  of  the  functional  properties  of  the 
AN/GSQ-183  Frequency  Control  Set,  which  is  intended  for  use  as 
an  interim  timing  subsystem  in  DCS. 

6. 2. 2.1  AN/GSQ-183  Frequency  Control  Set 

The  AN/GSQ-183  Frequency  Control  Set  uses  the  worldwide 
LORAN-C  transmissions  to  obtain  high  resolution  frequency  refer¬ 
ences  and  a  calibration  at  a  network  node.  The  system  generates 
a  1-MHz  output  signal  with  variability  of  a  microsecond  or 
better.  It  may  be  phase- locked  to  either  a  1-MHz  local  reference 
or  to  the  incoming  LORAN-C  signal. 

The  principal  components  are  shown  in  Figure  6.2.  The 
CV-3094  frequency  multiplier  consists  of  a  combiner  to  supply  a 
1-MHz  reference  to  the  LORAN  receiver.  This  signal  is  derived 
from  either  the  primary  or  secondary  RF  oscillators  (0  -  1632) 
using  an  automatic  selection  procedure  in  the  event  of  oscillator 
failure.  The  CV-3094  also  ensures  that  the  system  output  (1  MHz) 
is  slaved  to  the  best  reference  choice,  i.e.,  either  the  LORAN-C 
groundwave  or  the  alternate  local  reference.  When  neither  of 
these  signals  is  available,  each  RF  oscillator  is  put  into  a 
coast  mode  using  digital  control  signals,  as  shown.  This  unit 
and  the  oscillators  can  function  independently  of  the  LORAN 
receiver  using  alternative  1-MHz  frequency  sources  (sine  or 
square  wave). 

The  R-1776  receiver  is  a  solid-state  unit  designed  to  track 
the  LORAN-C  signals.  It  features  an  all-electronic  phase¬ 
tracking  servo  system,  with  20-ns  resolution. 

The  0  -  1632  oscillators  consist  of  a  voltage-controlled 
crystal  oscillator  with  digital  control  signals  and  provision 
for  manual  slewing  of  frequency  to  assist  in  acquisition.  The 
digital  servo  memory  retains  the  crystal  aging  rate  when  the 
loop  is  opened.  Finally,  the  CV-3093  frequency  multiplier  con¬ 
sists  of  three  identical  modules,  each  of  which  synthesizes 
96  kHz  and  1.2288  MHz  from  a  1-MHz  input  signal. 
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6. 2. 2. 2  AN/GSQ- 183  as  a  Backup  Frequency  Source 

The  VN/GSQ-183  output  may  be  visualized  as  a  frequency 
reference  with  excellent  long-term  stability  traceable  back 
to  an  atomic  standard  via  the  LORAN-C  path.  However,  quite 
severe  path  length  variations  are  evidenced,  and  this  is  equated 
with  unwanted  phase  and  frequency  jitter  on  the  output  signal. 

There  are  at  least  two  ways  of  integrating  the  AN/GSQ-183 
signals  into  the  Timing  Subsystem.  In  discussing  two  obvious 
possibilities,  we  shall  also  touch  on  the  subject  of  Independent 
Clock  operation  as  a  backup  mode. 

The  two  figures  which  follow.  Figures  6.3  and  6.4,  provide 
a  pictorial  statement  of  the  two  integration  choices  alluded  to 
above.  Some  explanation  of  the  symbols  and  notation  shown  is 
required  first.  Without  getting  into  the  subtleties  of  the 
complete  loop  design,  it  can  be  said  that  one  or  more  arrival 
time  measurements  from  terminated  links  will  be  processed  and 
combined  to  produce  a  clock  correction  signal.  In  its  simplest 
form,  this  is  a  clock  phase  error.  Figure  6.3(a),  for  example, 
has  three  terminated  links,  and  for  each  a  time  of  arrival  or 
phase  measurement  is  made  relative  to  the  local  node  clock. 

Figure  6.3(b)  is  the  transfer  function  block  diagram  equivalent 
of  (a).  The  inputs  are  represented  as  phase  variables  0j_,  02, 

63,  and,  after  being  compared  with  the  node  clock  phase  0q,  the 
differences  are  passed  to  the  control  loop  filter. 

The  LORAN-C  external  reference  is  represented  in  terms  of 
the  phase  of  the  originating  standard  0*,  and  a  propagation- 
induced  phase  perturbation  (i.e.,  noise)  0  .  The  total  is  desig¬ 
nated  ©l .  The  method  suggested  in  Figure  6.3  is  to  simply  treat 
the  external  LORAN  signal  as  another  equivalent  reference.  This 
is  implemented  by  generating  a  standard  TRP  with  repetition  rate 
of  one  per  second.  This  derived  timing  event  is  compared  with 
the  local  TRP,  and  the  resulting  time  interval  measurement  is 
used  in  the  clock  updating.  For  Time  Reference  Distribution,  the 
LORAN  TRP  would  not  be  used  explicitly  since  it  is  incapable  of 
providing  precise  time  data;  its  role  is  limited  to  stable 
relative  time  transfer.  Consider,  then,  the  situation  where 
none  of  the  links  can  provide  a  reliable  time  transfer  path. 

Then,  backup  operation  would  consist  of  maintenance  of  the  node 
clock  phase  so  that  the  relative  time  between  the  local  TRP  and 
the  LORAN  TRP  was  preserved.  The  loop  would  serve  to  lowpass 
filter  the  LORAN  time  variation;  thus,  the  loop  time  constants 
would  be  chosen  to  smooth  out  0n  in  Figure  6.3  and  hold  the  node 
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Figure  6.3  Integration  of  the  Interim  Subsystem  as 
an  Alternative  Reference 


clock  output  to  a  constant  relative  time  with  respect  to  the 
LORAN  standard.  With  this  form  of  backup,  there  is  no  stringent 
stability  requirement  for  the  local  standard  si*  it  is  effec* 
tively  locked  to  the  LORAN  atomic  standard. 

The  important  aspect  of  this  mode  is  that  the  LORAN-C  TRP 
signal  can  be  subjected  to  a  set  of  standard  consistency  and 
rejection  rules  in  the  same  way  that  link  termination  TRP's  are. 
Hence,  loss  of  the  LORAN  reference  because  of  transmission  or 
medium  disruptions  should  be  detectable.  Then,  the  local  station 
standard  resorts  to  stand-alone  operation  as  an  independent 
clock.  This  safety  mechanism  represents  a  second  level  of  backup 
for  the  node. 

The  other  approach,  depicted  in  Figure  6.4,  is  more  straight¬ 
forward,  and  perhaps  more  representative  of  circumstances  where 
the  station  standard  had  failed.  As  shown,  the  LORAN  1  MHz  is 
used  as  a  direct  substitute  for  the  station  standard.  While  link 
references  remain  valid  and  reliable,  the  servo  loop  can  operate 
with  them  as  shown.  The  output  signal  phase,  marked  by  the 
symbol  0q,  is  not  the  same  as  the  direct  LORAN  phase.  The  loop 
provides  highpass  filtering  of  the  LORAN  phase.  This  means  that 
any  low-frequency  fluctuations  in  0L  cause  the  loop  to  produce  a 
counteracting  phase  stepper  control  voltage;  therefore,  as  far  as 
0q  is  concerned,  there  is  no  effect.  The  node,  therefore,  de¬ 
pends  on  the  long-term  stability  of  the  incoming  link  references 
rather  than  on  the  LORAN-C  signal. 

When  the  node  must  operate  completely  without  the  link 
references,  it  would  make  sense  to  open  the  loop  to  obtain  the 
LORAN-C  long-term  stability  at  the  node.  This  is  done  quite 
simply  by  clamping  the  phase  stepper  control  voltage  at  zero. 

Then  the  microstepper  output  is  used  directly  by  the  node  synthe¬ 
sizer.  If  we  compare  this  open  loop  operation  of  the  configura¬ 
tion  in  Figure  6.4  with  that  shown  in  Figure  6.3,  it  is  seen  that 
while  they  are  basically  both  locked  to  the  LORAN  signal,  the 
latter  lowpass  filters  the  incoming  LORAN  signal  and,  in  addition, 
allows  the  node  to  revert  to  independent  clock  operation  without 
reconfiguration . 

The  recommendation  for  backup  node  operation  would,  there¬ 
fore,  be  to  use  the  "pseudo  link"  configuration  of  Figure  6.3 
except  when  the  station  standard  has  failed  completely,  in  which 
case  the  configuration  in  Figure  6.4  must  be  utilized,  either 
open  loop  or  closed  loop. 


In  concluding  this  section,  it  should  be  remarked  that, 
apart  from  sheer  physical  destruction  of  network  transmission 
equipment,  the  main  stress -inducing  element  will  probably  be  in 
the  nature  of  electronic  jamming.  While  the  jamming  may  be 
effective  enough  to  disable  the  normal  time  transfer  capabilities, 
it  should  be  realized  that  other  more  rugged  communications  op¬ 
tions,  such  as  spread-spectrum  transmissions,  will  often  be 
available.  With  the  maintenance  of  precise  time  at  both  ends  of 
a  link,  even  in  periods  of  disruption,  some  modest  level  of  com¬ 
munication  will  be  feasible.  This  points  out  the  importance  of 
establishing  backup  clock  synchronization  strategies  to  support 
such  emergency  communications  systems  if  they  exist  on  a  line. 

6 . 3  Network  Timing  Accuracy 

The  advantages  of  referencing  precise  time  are  numerous. 
Although  the  stated  accuracy  goal  for  DCS  is  2  ps  between  any 
two  nodes  in  the  network  [6.11],  apparent  accuracy  can  be  en¬ 
hanced  by  referencing  precise  time  at  one  or  more  nodes  in  the 
network.  This  section  is  concerned  with  the  availability  of 
precise  time  and  the  methods  by  which  to  use  it.  Finally,  a 
brief  review  of  Federal  Standard  1002  reveals  that  TS  compata- 
bility  currently  exists  and  compliance  is  already  "built  in"  to 
the  TS  prototypes. 

Some  of  the  potential  benefits  of  having  precise  time  trace¬ 
able  to  the  United  States  Naval  Observatory  (USNO)  or  National 
Bureau  of  Standards  (NBS)  are  directly  related  to  the  fact  that 
an  increased  distribution  of  universal  time  would  result.  This 
distribution  aids  in  the  general  availability  of  precise  time  to 
users  who  may  otherwise  have  difficulty  obtaining  it.  Networks 
which  disseminate  precise  time  traceable  to  USNO  or  NBS  have  more 
potential  to  interoperate  with  the  DCS,  resulting  in  increased 
survivability.  Also,  monitorability  is  enhanced  by  allowing  com¬ 
parison  of  precise  time  at  any  point  in  the  network  with  the 
external  source. 

In  summary,  the  benefits  of  precise  time  at  major  nodes  in 
a  network  are  listed  below: 

•  Aids  monitorability  and  self-reorganization 

•  Reduces  phase  disturbances  and  improves  frequency 
accuracy 

•  Improves  bit  synchronization  and  signal-to-noise 
performance 
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•  Improves  free-running  accuracy  following  failure 

•  Further  distributes  precise  time  to  potential  users 

•  Precise  time  is  useful  in  assisting  synchronization 
of  spread-spectrum  equipment  and  in  providing  syn¬ 
chronization  for  TDMA  networks  [6.10] 

•  Navigation  systems  could  use  the  DCS  timing  to 
increase  their  reliability  and  capability  [6.10] 

In  comparing  the  different  techniques,  the  major  question 
to  be  raised  is  the  size  of  the  frequency  step  between  primary 
and  backup  timing  sources.  The  bigger  the  frequency  step,  the 
faster  the  buffers  will  fill  or  empty  and  the  greater  the  proba¬ 
bility  of  bit  slip.  As  is  mentioned  in  several  places  in  this 
report,  it  appears  likely  that  a  Time  Reference  Distribution 
approach  would  be  ultimately  referenced  to  a  world-wide  time 
standard  such  as  UTC.  Similarly,  potential  backup  timing  sources, 
such  as  LORAN-C,  ^re  also  referenced  to  UTC.  Thus,  it  can  be 
anticipated  that  switch  to  a  backup  timing  source  in  a  network 
based  on  TRD  will  result  in  a  smaller  frequency  step  than  any  of 
the  other  techniques  since  both  primary  and  backup  systems  could 
be  referenced  to  the  same  ultimate  source  [6.10]. 

6.3.1  Precise  Time  Availability 

This  section  refers  to  universal  broadcasts  of  time  or 
frequency  signals  derived  from  a  precise  standard.  Included  in 
this  category  are,  for  example,  the  NBS  WWV  broadcasts,  LORAN-C 
and  Omega  navigation  systems,  TV  transmissions,  and  satellite 
broadcasts  [6.12]. 

The  utility  of  a  time /frequency  broadcast  in  network  syn¬ 
chronization  should  be  clear.  Each  control  node  in  the  network 
listens  in  on  the  timing  signal  and  synchronizes  its  own  clock 
accordingly.  There  are  three  issues  of  importance  in  considering 
the  various  possible  signals: 

(1)  Potential  timing  accuracy 

(2)  Geographical  coverage 

(3)  Propagation  disturbances 

The  WWV  broadcasts  originate  from  a  cesium  beam  standard 
maintained  by  NBS,  but  generally  speaking  the  accuracy  of  syn¬ 
chronization  that  can  be  obtained  is  typically  1  ms  [6.13],  which 
would  not  be  adequate  for  application  to  network  synchronization. 


Some  of  the  navigation  systems  offer  better  prospects. 
LORAN-C,  which  is  a  pulsed  100-kHz  carrier,  gives  accuracies 
of  1-2  /is  when  the  groundwave  components  can  be  received 
directly.  Skywave  reception,  a  necessity  for  intercontinental 
coverage,  results  in  accuracies  of  50  ^s,  which  is  not  really 
satisfactory  either.  Similarly,  the  Omega  VLF  signals,  being 
lower  in  frequency,  usually  p  ovide  accuracies  that  are  even 
worse,  typically  tens  of  ns.  Difficulties  arise  with  Omega  at 
locations  near  the  transmitter  where  groundwave-skywave  inter¬ 
ference  degrades  the  signal  quality. 

The  most  appealing  method  of  disseminating  time  or  frequency 
for  most  purposes  is  by  the  use  of  satellite  broadcasts.  With 
satellites,  one  obtains  the  combination  of  high  visibility  and 
high  bandwidth  transmissions,  which  are  not  usually  available 
with  a  ground-based  approach.  Two  distinct  strategies  can  be 
adopted  in  satellite  broadcast  systems,  i.e.,  one-way  and  two-way 
broadcasts.  In  a  one-way  broadcast,  the  user  receives  the  trans¬ 
mitted  signal  either  directly  from  an  on-board  satellite  clock 
or  from  a  ground-based  master  clock  via  the  satellite  acting  as 
a  time  transponder.  The  user  is  normally  provided  with  orbital 
or  path  length  and  rate  parameters  as  part  of  the  transmission. 

He  is  then  able  to  make  frequency  and/or  time  corrections  to  the 
incoming  signal. 

The  other  alternative  is  the  two-way  system,  which  requires 
no  transfer  or  orbital  parameters.  Instead,  the  method  requires 
an  exchange  of  timing  error  signals  between  the  master  and  slave 
stations.  The  two-way  timing  system  allows  each  slaved  node 
clock  to  compute  the  total  master/slave  path  delay  and,  by  suit¬ 
able  processing  of  error  signals,  synchronizes  itself  exactly 
with  the  originating  node.  With  a  satellite  broadcast,  the  two- 
way  system  is  quite  attractive  because  it  does  not  require  know¬ 
ledge  of  the  satellite  position  although,  clearly,  more 
transmitting  and  receiving  equipment  is  necessary. 

Examples  of  satellite  timing  systems,  operating  in  a  one-way 
node,  include  TRANSIT  [6.14]  and  TIMATION  [6.15].  Accuracies 
are  limited  to  a  few  microseconds.  One-way  experiments,  using 
satellites  simply  as  transponders,  have  been  carried  out  with 
ATS-1  and  ATS-3  [6.16],  LES-6,  and  TACSAT  [6.17] ,[6.18] .  In 
these  particular  experiments,  accuracies  as  high  as  10  ns  were 
obtained.  More  recently  [6.19],  one-way  transponding  experiments 
were  carried  out  using  WWV  as  a  standard,  and  the  ATS-3  satellite 
as  a  transponder.  The  resulting  accuracy  was  better  than  25  ps. 
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Although  the  emphasis  in  this  section  Is  seen  to  be  predomi¬ 
nantly  on  time  dissemination  rather  than  frequency  dissemination, 
it  should  be  pointed  out  that  the  slave  nodes  do  not  require 
exact  time  synchronization  to  function  correctly  in  a  digital 
communications  network.  For  example,  the  incoming  clock  can 
differ  from  the  node  clock  by  a  fixed  phase  (and,  hence,  path 
delay)  without  any  adverse  consequences.  Clearly,  it  is  the 
magnitude  of  dynamic  phase  excursions  which  affect  buffer  over¬ 
flow/underflow  statistics,  and  these  phase  variations  are  derived 
from  an  integral  of  the  instantaneous  frequency  difference.  Two- 
way  clock  error  exchanges,  therefore,  seem  of  less  value  than  a 
knowledge  of  the  rate  of  change  in  satellite  position,  i.e.,  the 
Doppler  shift  perceived  at  the  receiver.  If,  for  example,  the 
path  delay  via  a  satellite  was  changing  slowly  with  time  (e.g., 
at  a  constant  rate),  the  two-way  clock  correction  scheme  would 
maintain  precise  synchronization  between  the  master  and  slave 
clocks  regardless  of  the  path  delay.  However,  the  incoming  data 
rate  would  then  be  slightly  higher  than  the  transmitted  rate, 
reflecting,  of  course,  the  steady  depletion  of  bits  stored  in 
the  transmission  "buffer",  and  the  slave  node  buffer  would  fill 
at  a  corresponding  rate.  In  effect,  the  total  number  of  bits 
stored  in  the  medium  plus  node  buffer  remains  invariant  with 
perfectly  synchronized  clocks. 

Precise  time  may  be  made  available  to  DCS  equipment  without 
penalizing  development  or  implementation  costs.  Survivability, 
however,  becomes  a  factor  with  ground-based  systems,  such  as 
LORAN-C,  where  global  time  signals  are  provided  by  relatively 
few  stations.  Also,  the  sheer  physical  size  of  the  transmitting 
antenna  structure  makes  it  an  easy  target  for  enemy  fire.  Over¬ 
all,  the  benefits  of  precise  time  cannot  be  denied,  and  its  use 
is  recommended.  In  this  light,  the  use  of  satellites  becomes 
even  more  desirable  since  satellite  transferred  timing  is  also 
derived  from  ground-based  sources. 

6.3.2  Dissemination  of  Precise  Time 

Precise  time  can  be  distributed  by  any  system  that  is 
double-ended  and  has  directed  control.  Some  of  the  Improved 
TRD  features  are  actually  designed  to  further  enhance  this  idea, 
but  the  aforementioned  two  requirements  are  the  basic  ones 
needed.  For  example,  phase  reference  combining,  as  described 
in  [6.9]  and  [6.10],  further  improves  the  accuracy  of  precise 
time  dissemination. 
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It  has  been  demonstrated  in  [6.9]  that  the  inclusion  of 
precise  time  does  not  degrade  the  performance  of  the  system.  In 
fact,  the  only  drawbacks  are  slightly  increased  channel  overhead 
and  processor  computational  burdens,  but  these  may  be  deemed 
insignificant . 

In  [6.9],  a  double-ended  systenrwith  node  B  slaved  to  node  A 
is  assumed.  Without  precise  time,  the  system  Phase-Locked  Loop 
(PLL)  deriving  the  nodal  frequency  at  node  B  would  be  driven  by 
the  estimated  phase  error  similar  to  that  used  in  a  precise  time 
system  such  as  TRD[6.20].  The  measured  phase  error  has  an  ambi¬ 
guity  of  some  arbitrary  number  of  cycles.  However,  once  the 
phase  error  difference  between  the  nodes  and  its  relationship  to 
the  times  of  the  clocks  are  defined,  then  the  measured  phase 
error  is  a  constant,  and  any  changes  in  clock  times  cause  corres¬ 
ponding  changes  in  measured  phase  error.  Assuming  equal  path 
delay  in  both  directions,  the  correction  term  (consisting  of 
clock  time  differences)  is  equal  to  the  actual  timing  error.  To 
convert  a  normal  double-ended  system  into  a  precise  time  system, 
we  must  include  an  additional  term  comprised  of  the  phase  differ¬ 
ence  correction  term  as  does  Improved  TRD  [6.20].  This  correc¬ 
tion  term  is  a  constant  and  can  be  estimated  through  infrequent 
transmissions  of  a  time-of-day  mark.  Thus,  there  is  no  perfor¬ 
mance  penalty  associated  with  the  inclusion  of  precise  time  since 
the  nodal  frequencies  are  determined  only  as  they  would  be  in  a 
system  without  precise  time.  Phase  and  frequency  accuracy  are 
the  same.  The  dissemination  of  precise  time  results  in  improved 
monitorability .  When  reorganization  is  used,  the  improved  moni- 
torability  will  result  in  faster  response  time  for  reorganization 
which,  in  turn,  improves  survivability.  In  addition,  future 
users  will  have  an  alternate  source  of  precise  time  readily 
available. 

6.3.3  Compliance  with  Federal  Standard  1002 

The  DCS  timing  subsystem  must  comply  with  Federal  Standard 
1002  to  facilitate  interoperability  between  telecommunications 
facilities  and  systems  of  the  Federal  Government  without  mea¬ 
surably  reducing  the  survivability  or  flexibility,  or  increasing 
equipment  or  system  complexity  or  cost. 

Federal  Standard  1002  states  that  "the  time  and  frequency 
reference  information  utilized  in  applicable  Federal  Government 
telecommunications  facilities  and  systems  shall  be  referenced  to 
(known  in  terms  of)  the  existing  standards  of  time  and  frequency 
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maintained  by  the  U.S.  Naval  Observatory,  UTC  (USNO) ,  or  the 
National  Bureau  of  Standards  UTC  (NBS)".  Furthermore,  ‘'the 
accuracy  of  this  time  and  frequency  information  with  respect  to 
UTC  (USNO)  or  UTC  (NBS)  shall  be  commensurate  with  the  individual 
system  design  and  interface  requirements".  The  above  statements 
may  be  interpreted  as  specifying  that  the  DCS  synchronization 
subsystem  reference  universal  precise  time  [UTC  (USNO)  or  UTC 
(NBS)]  and  to  provide  the  vehicle  with  which  the  reference  may 
be  distributed  throughout  the  network.  In  addition,  how  accu¬ 
rately  these  references  are  maintained  depends  on  the  capability 
of  the  synchronization  system  and  limitations  imposed  by  inter¬ 
facing  requirements. 

The  current  design  of  the  Timing  Subsystem  affords  easy  com¬ 
pliance  to  the  above-mentioned  Federal  Standard.  Each  TS  proto¬ 
type  can  reference  any  one  of  three  auxiliary  external  sources  in 
1-pps  format.  Accuracy  limitations  imposed  are  dictated  by  the 
resolution  of  the  time  interval  measurement  system  which  is  10  ns 
for  the  current  design.  This  is  especially  applicable  for  the 
case  of  directed  control  where  only  one  node  need  reference  an 
external  precise  time  reference.  This  (master)  node  is  then  con¬ 
figured  to  follow  the  reference  via  its  internal  PLL  tracking 
system.  Slave  nodes  then  indirectly  follow  the  universal  time 
reference.  Other  aspects  of  network  synchronization  are  not 
directly  impacted  by  the  Standard.  These  include  self  organiza¬ 
tion,  independence  of  clock  error  measurement  and  correction,  and 
phase  reference  combining  features  [6.9]. 

One  of  the  more  compelling  reasons  for  recommending  the  DCS 
reference  precise  time  is  to  comply  with  Federal  Standard  1002. 
This  is  not  seen  as  an  obstacle  but  rather  as  a  desirable  feature 
in  any  large  network,  since  compliance  to  the  Standard  does  not 
present  any  hardship.  Presently,  DCA  is  committed  to  provide 
DSCS  terminals  which  are  slaved  to  precise  time  via  satellite 
time  transfer. 

6 . 4  Timing  Technique  Evaluation 

It  would  be  exceedingly  difficult  to  summarize  the  perfor¬ 
mance  of  each  candidate  timing  technique  by  a  direct  comparison 
of  features.  Correspondingly,  it  is  not  just  a  matter  of  saying 
that  candidate  A  is  better  than  candidate  B;  each  technique  has 
its  own  merits  and  none  of  them  are  considered  superior  in  all 
evaluation  categories.  However,  during  the  course  of  this  pro¬ 
gram,  it  became  clear  that,  at  least  from  observed  network 
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synchronization,  the  performance  of  each  technique  would  be, 
within  given  measurement  capabilities,  nearly  identical.  This 
is  due  primarily  to  the  fact  that  such  a  small  (three-node) 
network  was  configured.  Although  it  was  not  possible  to  thor¬ 
oughly  analyze  the  advantages  and  tradeoffs  associated  with  each 
of  the  synchronization  techniques  as  applied  to  large  networks, 
much  insight  was  gained  as  to  the  implementation  difficulty  and 
"ease  of  use"  once  in  the  field.  The  reader  is  referred  to  [6.9] 
and  [6.10]  for  discussions  of  timing  technique  performance  in 
larger  networks.  Much  of  the  comparison  for  this  program  was 
carried  out  at  the  network  level;  that  is,  link  characteristics 
were  not  investigated  in  detail  as  a  previous  CNR  effort  [6.5] 
evaluated  TROPO  and  LOS  link  performance  when  used  to  support  a 
timing  function. 

For  a  review  of  each  of  the  timing  technique  candidates, 
see  Section  5.2;  they  will  not  be  repeated  here.  To  begin  the 
comparison,  the  reader  is  reminded  that  each  of  these  techniques 
was  implemented  with  identical  Timing  Subsystem  hardware;  no 
modifications  were  required  when  specifying  different  timing 
algorithms  since  they  existed  totally  in  software.  Each  Timing 
Subsystem  contains  a  precision  time  interval  counter,  FIFO,  and 
level-1  MUX,  thus  eliminating  the  need  for  retrofit  or  redesign 
of  service  channel  MUX.  Some  problems  occur  when  extracting 
framing  pulses  from  existing  equipment  (e.g.,  modems)  for  SDLC 
TRIP  transmission,  but  these  problems  are  not  difficuli-  to  over¬ 
come.  The  next  section  addresses  some  of  these  issues. 

Much  of  the  field  effort  was  spent  getting  on-the-air  and 
keeping  the  link  equipment  operational.  Thankfully,  the  Timing 
Subsystems  proved  to  be  extremely  reliable  and  not  one  failure 
was  experienced  (a  bad  batch  of  power  supplies  and  a  lightning 
strike  slowed  progress  somewhat)  once  in  the  field.  As  ex¬ 
plained  in  Sections  3  and  5,  only  a  single-ended  tandem  network 
was  possible,  dictated  by  the  equipment  that  was  working.  The 
lack  of  a  double-ended  link  precluded  the  use  of  Time  Reference 
Distribution  as  it  was  intended  to  be  used.  This  is  especially 
true  for  the  Improved  Time  Reference  Distribution  technique  in 
which  phase  reference  combining  is  introduced  to  enhance  system 
accuracy  [6.20].  Double-endedness  is  a  necessary  feature  to 
preserve  precise  time  and  to  eliminate  path  delay  factors  across 
a  link.  However,  for  the  sake  of  an  exercise,  the  original  Time 
Reference  Distribution  scheme  [6.21],  as  modified  by  the 
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Clarkson  College  study  [6.22],  was  run  successfully  when  tried 
with  single-ended  transfers  (see  Section  5.5.4  for  details  of 
this  experiment) . 

Much  of  the  experimental  program  was  spent  investigating 
the  performance  of  a  simple  or  tandem  Master/Slave  scheme.  Al¬ 
though  only  single-ended  transfers  were  possible,  the  overall 
simplicity  of  this  technique,  as  a  main  or  fallback  system, 
should  not  be  overlooked.  This  has  been  emphasized  by  several 
studies  [[6.5],  [6.9],  and  [6.10],  for  example),  and  is  empha¬ 
sized  here  again.  Other  than  Independent  Clocks,  Master/Slave 
required  the  least  software  and  channel  overhead.  However,  its 
merit  as  a  wartime  system  is  questioned,  especially  to  its  lack 
of  self-organization  capabilities.  Clearly,  some  form  of  clock 
control  distribution  would  be  desired,  or  perhaps  an  adaptive 
reorganization  scheme  such  as  that  employed  by  the  Canadian 
Dataroute  [6.23] ,[6.24] . 

Such  features  are  fundamental  to  the  Improved  TRD  technique 
which  has  the  qualities  most  often  associated  with  a  survivable 
system.  As  listed  in  Section  5.2.4,  these  are: 

•  Directed  control 

•  Double-endedness 

•  Independence  of  clock  error  measurements  and 
corrections 

•  Self-organizing 

•  Phase  reference  combining 

Without  going  into  further  detail  about  these  concepts,  the 
reader  is  referred  to  [6.20]  and  [6.25]  -  [6.28]  which  offer  more 
complete  studies. 

Mutual  Synchronization  offers  very  little  over  other  tech¬ 
niques  except  to  distribute  references  to  all  nodes  in  the 
network.  Reference  [6.9]  demonstrates  that  MS  is,  at  best,  a 
somewhat  unstable  system  with  system-wide  errors  compounding  if 
a  nodal  clock  goes  bad.  Mutual  systems  that  reference  a  single 
master  may  have  added  merit,  but  they  will  not  be  discussed  here. 
Laboratory  tests  only  strengthened  our  position  that  mutual  con¬ 
trol  systems  should  not  be  considered  for  large  networks  where 
many  closed  loop  timing  paths  and  instability  would  most  cer¬ 
tainly  be  the  result.  In-house  tests  confirmed  that  the  poten¬ 
tial  for  network  instability  is  great  -  even  in  smaller  systems. 
Careful  prefiltering  of  calculated  clock  error  must  be  observed. 


In  short,  it  is  suggested  that  Mutual  Synchronization  not  be 
considered  for  use  in  DCS  as  a  wartime  system  because  the  other 
techniques,  or  perhaps  combinations  of  them,  offer  superior  ad¬ 
vantages  over  the  mutual  system,  especially  in  large  networks. 

In  contrast,  the  simplest  method,  that  of  Independent  Clock, 
has  been  shrugged  off  by  some  as  totally  unacceptable  and  by 
others  as  a  technique  worth  some  careful  consideration.  The 
concept  of  Independent  Clock  has  some  merit,  especially  when  one 
considers  the  precision  frequency  standards  and  large  FIFO 
buffers  available  (to  absorb  frequency  offsets).  With  high- 
quality  equipment,  buffer  resets  could  be  anticipated  only  after 
several  days,  for  the  worst  case.  Also,  installation  costs  are 
moderate,  but  a  host  of  processing  and  loop  control  equipment 
could  be  eliminated.  In  the  effort  of  maximally  utilizing  tvail- 
able  time,  CNR  did  not  test  Independent  Clocks  in  the  field, 
although  it  is  not  difficult  to  predict  performance  via  a  few 
calculations  or  a  simple  simulation.  Instead,  it  is  stated  that 
if  it  were  not  for  a  periodic  interruption  of  traffic,  this 
technique  would  most  likely  be  given  more  credit. 

Finally,  it  should  be  clear  that  recommendations  are  nar¬ 
rowed  down  to  Master/Slave  and  TRD  choices.  Without  choosing 
either  one  it  should  be  stated  that  Improved  TRD  was  not  com¬ 
pletely  exercised  on  the  Test  Bed  network  due  to  the  lack  of 
transmission  facilities  to  provide  double-ended  transfers.  How¬ 
ever,  it  is  clear  that  Improved  TRD  has  been  designed  to  incor¬ 
porate  all  of  the  "suggested  attributes"  [6.26] ,[6.29]  for  timing 
in  the  future  DCS. 

The  distinctive  feature  of  Time  Reference  Distribution  is 
that  a  network  with  reasonably  stable  clocks  at  each  node  is 
forced  to  take  corrective  action  at  these  nodes  on  the  basis  of 
perceived  clock  errors  which  are  continually  passed  around  the 
network  from  node  to  node.  As  in  the  previous  discussion 
(Section  5.2.4  and  Appendix  A),  all  nodal  clocks  are  rank-ordered 
and  the  timing  of  each  incoming  link  is  compared  with  the  local 
clock  in  the  derivation  of  a  timing  error  for  that  link.  This 
error  signal  is  transmitted  back  to  the  originating  node  which 
is  likewise  computing  its  own  nodal  clock  error  for  the  link. 

With  the  ensuing  exchange  of  clock  error  signals  between  both 
ends  of  the  link,  each  of  the  two  nodes  has  the  capability  of 
computing  the  true  nodal  clock  time  difference  with  virtually 
no  dependence  on  path  delay. 
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The  selection  rules  ensure  that  no  closed  loops  are  sec  up, 
mainly  because  of  the  reduction  in  rank  which  occurs  for  a  clock 
signal  as  it  is  passed  through  additional  links.  The  principal 
advantages  of  the  Time  Reference  Distribution  technique  over 
other  synchronization  methods,  such  as  independent  stable  clocks 
or  Mas ter/S lave  configurations,  can  be  stated  as  follows: 

(1)  The  most  accurate  clock  available  in  the  network 
always  governs  the  complete  system. 

(2)  Self -monitoring  of  node  performance  is  enhanced 
by  the  availability  of  the  various  link  time 
error  signals. 

(3)  Propagation  of  medium  delay  variation  is  minimized 
by  the  cancellation  of  path  delay  in  the  formula¬ 
tion  of  error  signals. 

(4)  Successful  system  operation  is  not  dependent  upon 
the  continued  operation  of  any  one  node.  In  the 
event  of  failure  of  the  current  master  node,  the 
system  reorganizes  to  find  an  alternative  master 
substitute.  The  resulting  transition  would  be 
smooth  provided  the  local  clock-circuit  time  con¬ 
stant  was  long  compared  with  the  time  to  reorganize 
and/or  sufficiently  large  data  buffers  were  avail¬ 
able  to  absorb  transient  effects. 

(5)  Network  growth  and  change  are  easily  accommodated 
(e.g.,  upgrading  of  clocks  at  the  nodes). 

With  the  additional  refinements  built  into  the  proposed  Time 
Reference  Distribution  technique  [6.20],  there  is  ample  evidence 
that  TRD  should  perform  well  in  a  large  network,  especially  when 
referenced  to  universal  time  (its  intended  application) [6. 27] . 

It  should  be  recognized  that  recommendations  are  based  on 
several  criteria.  First,  much  of  the  initial  reaction  to  each 
technique  was  formed  when  implementing  each  in  software.  At  that 
time,  the  ease  of  putting  the  algorithms  into  code  was  discovered 
as  well  as  any  problems  with  interpretation.  For  this  comparison 
Improved  TRD  was  more  difficult  to  interpret  and  implement  with 
TRD,  Mutual  Sync,  Master/Slave,  and  Independent  Clocks  following. 
Appendix  A  contains  a  review  of  the  TRD  techniques,  including 
flowcharts  of  the  algorithms. 


6-33 


Secondly,  laboratory  tests  revealed  the  characteristics  of 
each  technique  at  the  network  level.  Several  experiments  were 
run  demonstrating,  for  instance,  the  general  behavior  of  directed 
control  systems,  including  the  self-organizational  capabilities 
of  TRD.  Each  candidate  technique  appeared  to  work  well  (within 
the  bounds  of  the  test  equipment  limitations) .  Mutual  Sync 
proved  to  be  somewhat  temperamental,  and  even  with  additional 
error  prefiltering,  never  performed  satisfactorily  (see  Section 
5.2.3).  Overall,  the  TRD  techniques  seemed  to  be  most  promising 
although  defining  and  inputting  nodal  parameters  was  rather 
tedious.  In  contrast,  Independent  Clocks  and  Master/Slave  re¬ 
quired  minimal  setup  time. 

Finally,  field  tests,  though  limited,  further  reinforced  the 
preferences  formed  in  the  laboratory.  No  surprises  were  found 
in  TS  behavior  and,  in  general,  the  directed  control  (including 
TRD)  experiments  performed  admirably. 

In  summarizing  the  capabilities  of  the  techniques,  it  should 
be  evident  that  each  will  work  when  applied  properly.  Although 
it  would  be  difficult  to  endorse  a  particular  technique  in  light 
of  a  somewhat  limited  experimental  program,  enough  insight  has 
accumulated  to  allow  a  certain  degree  of  acceptance  with  previous 
findings;  for  example,  it  seems  apparent  that  any  one  of  the  can¬ 
didate  techniques  is  applicable  at  the  highest  level  of  the  DCS 
hierarchy  (with  great  reservations  about  Mutual  Synchronization) . 
Also,  due  to  the  connectivity  of  the  network,  the  lowest  level 
nodes  should  be  slaved  to  their  corresponding  intermediate  or 
highest  level  node.  This  is  due  to  the  limited  connectivity  that 
characterizes  the  lowest  level  nodes  [6.10].  Therefore,  it  is 
felt  that  Master/Slave  is  a  good  choice  for  the  lowest  levels  of 
network  hierarchy,  while  the  reorganizational  and  phase  accuracy 
traits  of  TRD  are  well-suited  for  use  on  the  higher  levels  of 
network  topology  where  survivability,  accuracy,  and  monitor- 
ability  are  the  essential  ingredients. 

6.5  Future  Work  in  Network  Synchronization 

Throughout  the  test  program,  the  Timing  Subsystem  proved  to 
be  reliable,  efficient  equipment.  Every  precaution  was  observed 
to  build  a  system  capable  of  meeting  the  interfacing  require¬ 
ments  for  both  current  and  future  DCS  equipment  while  incorpora¬ 
ting  programmability.  Based  on  experience  gained  in  this 
contract,  it  is  possible  to  make  recommendations  concerning 
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the  general  approach  to  network  synchronization  and  the  adoption 
of  a  timing  technique  by  the  DCS.  This  section  labels  some  of 
the  more  practical  obstacles  encountered  in  the  field  as  well  as 
suggestions  for  an  advanced  development  model  based  on  the 
current  TS  design  and  operating  experience  in  a  network 
environment . 

Without  a  doubt,  before  a  strong  synchronization  technique 
recommendation  can  be  made,  much  more  testing  in  a  larger  (say, 
at  least  six  nodes)  Test  Bed  should  be  conducted.  Surely,  if 
such  a  task  were  to  be  undertaken,  based  on  experience,  network 
synchronization  experiments  should  be  limited  to  Master/Slave  and 
Improved  TRD,  or  some  combination  thereof.  Master/Slave  could 
provide  the  simple  reliable  base  for  timing  distribution 
(directed  control)  while  the  self-organizing  and  increased 
accuracy  features  of  Improved  TRD  may  be  used  to  enhance  this 
already  field-proven  technique  [6.28]. 

One  of  the  major  advantages  of  directed  control  is  the 
potential  cost  savings  associated  with  the  use  of  quartz  stan¬ 
dards  in  place  of  atomic  standards  at  the  hundreds  of  less 
significant  nodes  provide  justification  for  development  of  the 
Timing  Subsystem  to  accommodate  them.  Throughout  this  report 
we  have  endeavored  to  highlight  design  features  which  will  en¬ 
hance  the  performance  of  quartz  standards .  An  example  is  the 
attention  we  have  given  to  automatic  tracking  of  the  local 
standard  frequency  drift  and  offset  that  occur  with  aging.  It 
should  be  realized  that  the  DCS  network  can  be  expected  to  run 
for  very  long  periods  with  a  single  high-quality  master  standard, 
and  such  intervals  provide  the  subservient  nodes  an  opportunity 
to  qualitatively  assess  the  offset  and  drift  of  their  own  station 
standards.  Then,  in  times  of  network  stress  when  independent 
clock  operation  may  be  called  for,  this  knowledge  can  be  used  to 
compensate  the  local  standard  with  better  results  than  without 
clock  correction. 

A  major  portion  of  the  field  effort  for  this  program  was 
spent  trying  to  get  'on-the-air' .  The  sheer  quantity  of 
equipment  associated  with  even  a  small  network  requires  a  truly 
dedicated  technical  staff  to  maintain  and  operate  it,  especially 
at  geographically  distant  sites.  For  larger  networks,  the 
staffing  burden  becomes  even  greater.  Currently,  surveys  have 
shown  that  networks  with  relatively  simple  timing  schemes  have 
required  the  skills  of  engineering- level  personnel  to  deploy  and 
operate  the  associated  timing  equipment  because  the  lack  of  a 


ready  supply  of  digitally-oriented  field  personnel  was  noted 
[6.9].  However,  the  subsystem  type  or  its  features  had  little 
influence  on  deployment  difficulties.  Therefore,  overall  sim¬ 
plicity  and  high  monitorability  should  be  considered  desirable 
traits.  Future  timing  subsystems  should  provide  automatic  self¬ 
monitoring  and  be  able  to  warn  personnel  of  an  approaching  fault 
condition.  This  may  be  implemented  with  control  software  as 
necessary,  although  signal  or  power  supply  monitoring  may  require 
additional  hardware  considerations.  Hardware  status  indicators 
should  be  available  to  site  personnel  at  all  times. 

The  area  of  equipment  reliability,  although  briefly  ad¬ 
dressed,  is  not  considered  a  problem  as  indications  have  shown 
that  timing  and  synchronizing  equipment  is  highly  reliable.  This 
was  substantiated  by  zero  failures  in  the  field  after  periodic 
operation  over  a  span  of  about  six  months.  Frequency  standards 
also  are  quite  reliable  and  should  serve  to  enhance  overall 
network  reliability. 

Additionally,  further  development  in  Timing  Subsystem  soft¬ 
ware  could  provide  added  features .  Since  the  TS  contains  a 
general-purpose  microcomputer  capability,  programming  options 
are  virtually  unlimited.  It  is,  however,  important  to  rigorously 
define  not  only  the  nodal  processing  and  data  flow  techniques 
but  also  to  specify  the  nature  and  extent  of  interaction  with  the 
operator.  It  makes  sense  to  utilize  software  capabilities  to  the 
fullest  extent;  feedback  to  a  node  operator  perhaps  being  one  of 
the  more  urgent  possibilities.  Powerful  monitor  software  would 
greatly  improve  operability  and  lessen  the  need  for  highly  expe¬ 
rienced  or  engineering- level  personnel  at  each  installation. 
System  status  directly  output  to  the  user  could  serve  as  an  early 
warning  system  for  pending  problems.  Overall  "ease  of  use"  would 
be  greatly  improved  by  user  friendly,  English-like  dialog.  Sig¬ 
nificant  changes  in  the  node  software  may  be  incorporated  without 
jeopardizing  the  hardware  design  or  capabilities. 

Clearly,  network  testing  on  a  larger  scale  than  possible 
during  this  program  would  then  provide  additional  information 
necessary  to  confirm  these  preliminary  findings.  The  Timing 
Subsystem  is  certainly  capable  of  operating  in  a  larger  network. 
Link  redundancy  must  be  thought  of  as  an  important  factor  in  a 
larger  network;  that  is,  how  will  a  timing  candidate  cope  with 
a  closed  loop  timing  path?  It  is  not  practical  to  resolve  this 
question  here,  but  experimentation  with  a  technique  such  as 
Improved  TRD  could  provide  insight  to  this  problem.  Another 
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area  worth  at  least  some  Investigation  is  the  effect  of  clock 
control  loop  bandwidth  on  the  performance  of  network  synchroni¬ 
zation.  Although  the  TS  incorporates  a  multi-stage  PLL  algorithm 
with  programmable  parameters,  the  testing  schedule  did  not  allow 
experimentation  in  this  area,  and  it  is  suggested  that  future 
work  address  this  subject.  For  the  sake  of  the  test  plan,  loop 
parameters  similar  to  those  in  the  CNR  effort,  under  Contract 
F30602-76-C-0347,  were  used  because  they  were  chosen  as  effective 
for  the  communications  links  employed  (see  [6.5]). 

The  Timing  Subsystem  development  program  has  sufficiently 
demonstrated  that  communication  link  termination,  clock  control 
and  correction,  network  synchronization,  and  performance  assess¬ 
ment  capabilities  can  be  implemented  using  a  single  micro- 
cessor-controlled  device.  Furthermore,  there  is  ample  evidence 
that  the  current  design  can  support  the  interfacing  and  pro¬ 
cessing  requirements  for  an  advanced  development  model  well  into 
the  foreseeable  future.  It  has  been  recommended  that  the  next 
generation  Timing  Subsystem  incorporate  improved  operating  soft¬ 
ware  while  maintaining  the  basic  design  and  programmability  of 
the  system.  As  many  options  as  possible  have  been  left  open  to 
enhance  flexibility  and  to  insure  that  future  applications  can 
be  handled  without  the  need  for  large  expenditures  or  redesign. 

Recently,  a  set  of  desirable  attributes  has  been  suggested 
for  adoption  into  the  digital  DCS  to  increase  the  suitability 
for  wartime  use  [6.27].  These  attributes  address  both  the  hard¬ 
ware  associated  with  the  timing  system  and  the  desired  features 
of  the  synchronization  technique  itself.  The  main  approach  is 
to  provide  a  more  reliable,  survivable  system  capable  of  sup¬ 
porting  a  timing  function  in  a  hostile  environment.  Including 
these  attributes  into  the  Timing  Subsystem  is  well  within  the 
capabilities  of  hardware  and  software.  Many  of  these  features 
are  inherent  :'n  the  design  of  the  system,  while  others  require 
incorporation  into  the  timing  technique  currently  residing  in 
software.  Overall,  the  current  Timing  Subsystem  design  provides 
a  firm  base  for  development  of  future  subsystems  intended  to 
possess  the  fundamental  characteristics  considered  essential 
ingredients  of  the  DCS  or  other  communications  systems. 
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APPENDIX  A 


SOFTWARE  IMPLEMENTATION  OF  TIME  REFERENCE 
DISTRIBUTION  TECHNIQUES 


This  presentation  of  TRD  interpretation  is  intended  to  be 
used  as  a  guide  to  the  techniques  employed  by  CNR  to  implement 
these  algorithms  in  software.  The  organization  is  as  follows: 

•  The  first  section  describes  TRD  as  presented  in  [1] 
and  the  Clarkson  College  Report  [2]  which  includes  an 
obsolete  data  handler  not  specified  in  the  original 
TRD  proposed  by  H.  Stover  [1],[3]. 

•  The  second  section  describes  an  Improved  TRD  Tech¬ 
nique  [4]  which  includes  several  refinements  to  enhance 
monitorability  and  improve  accuracy. 

Each  technique  is  presented  along  with  flowcharts  to  aid 
in  visualizing  the  sequence  of  events  within  the  algorithms. 

1.0  Time  Reference  Distribution 


This  method  of  Time  Reference  Distribution  is  a  modified 
version  of  self-organization  which  has  been  shown  to  organize 
a  network  as  well  as  reorganize  it  after  a  disturbance.  It 
ensures  the  automatic  selection  of  the  highest  ranking  node  as 
the  master  node  as  well  as  transmission  of  information  to  each 
node  from  this  master  node  by  a  path  of  minimum  demerit.  In 
addition,  once  the  network  has  been  organized,  each  node  knows 
its  position  with  respect  to  the  master  node  in  the  hierarchy 
structure  along  which  information  is  disseminated  throughout 
the  network. 

TRD  consists  of  a  simple  set  of  rules  which  permit  each 
node  to  choose  its  time  reference  from  that  particular  neigh¬ 
boring  node  which  will  result  in  the  local  node  getting  its 
ultimate  time  reference  from  the  best  clock  in  the  network  over 
the  best  transmission  path  from  that  ultimate  reference  clock. 
When  failures  occur  somewhere  in  the  network,  problems  can 
result  unless  sufficient  time  is  allowed  for  obsolete  informa¬ 
tion  to  be  swept  from  the  network  following  a  failure.  Since 
information  about  a  failure  progresses  only  one  node  farther 
from  the  failed  node  (or  the  node  associated  with  a  failed  link) 
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for  each  information  transfer,  many  information  transfers  might 
be  required  before  obsolete  information  is  replaced  [2]. 

This  problem  resulted  because  nodes  were  permitted  to  make 
decisions  based  on  obsolete  information  following  a  failure  in 
the  system.  The  problem  can  be  avoided  by  assuring  that  any 
node  which  must  change  to  an  alternate  reference  as  the  result 
of  a  failure  in  the  network  waits  a  sufficient  length  of  time 
for  obsolete  information  to  be  replaced  in  the  adjacent  node 
f_om  which  the  reference  will  be  taken.  Several  different 
approaches  are  possible  for  determining  the  number  of  infor¬ 
mation  exchanges  to  wait  before  accepting  an  adjacent  node  as  a 
reference.  For  the  type  of  operation  of  interest,  this  does 
not  appear  to  be  a  critical  factor  so  long  as  the  waiting  time 
is  at  least  great  enough  [2], 

The  TRD  process  is  an  implementation  of  the  algorithm  speci¬ 
fied  in  the  Clarkson  College  Report  [2],  The  algorithm  is  iden¬ 
tical  to  the  H.  Stover  Report  [3]  except  that  Dr.  Perreault's 
proposed  solution  to  the  obsolete  data  problem  is  implemented. 

See  Figure  1. 

The  first  part  of  the  algorithm  performs  the  selection 
of  the  best  available  reference  according  to  the  rules  speci¬ 
fied  in  both  reports.  These  rules  applied  in  order  are: 

Rule  1:  A  node  initially  entering  the  network  will 
temporarily  reference  its  own  clock  until  a 
better  selection  is  made. 

Rule  2:  Whenever  the  link  or  neighboring  node  which 
a  node  uses  for  its  immediate  time  reference 
fails,  the  node  will  temporarily  reference  its 
own  clock  until  an  alternate  selection  is  made. 

Rule  3:  If  a  neighboring  node  being  used  as  the  imme¬ 
diate  reference  should  have  a  change  in  its 
ultimate  reference  to  one  of  a  lower  rank,  the 
local  node  will  temporarily  reference  its  own 
clock  until  an  alternate  selection  (which  could 
be  the  same  one  again)  is  made. 

Rule  SI:  Select  the  neighboring  node  that  has  the  best 
Nl,  if  inconclusive  apply  S2 
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Rule  S 2 :  Select  the  neighboring  node  of  those  selec¬ 
ted  by  Si  that  has  the  best  N2,  if  inconclusive 
apply  S3 

Rule  S3:  Select  the  neighboring  node  of  those  selected 
by  S2  that  has  the  best  N3 

The  actual  reference  switch  cannot  be  done  with  only  these 
rules  or  the  problem  of  obsolete  data  existing  in  the  network 
during  reorganization  can  occur.  An  update  counter  can  be  pro¬ 
vided  at  each  node  and  used  as  parameter  N4  in  addition  to  para¬ 
meters  Nl,  N2  and  N3. 

The  solution  to  this  problem  proposed  in  the  Clarkson 
College  Report  involves  the  application  of  three  more  rules  in 
order  using  the  local  and  received  update  counter  (N4)  values: 

Rule  Dl :  When  the  tentative  best  reference  is  the 
node  itself,  the  update  counter  is  decre¬ 
mented  by  1  (to  a  minimum  of  0). 

Rule  D2:  If  the  received  update  count  from  the  tenta¬ 
tive  choice  is  smaller  than  the  local  update 
count,  the  tentative  reference  selection  is 
confirmed  and  the  received  update  count  is 
incremented  by  1,  and  thereafter  used  at 
that  node. 


Rule  D3:  Otherwise,  the  node  uses  itself  for  a  refer¬ 
ence  and  increments  its  own  update  counter 
by  1. 

The  reader  is  referred  to  References  [1],  [2],  [3]  for 
further  discussion  of  this  algorithm. 


Improved  Time  Reference  Distribution 


The  Improved  Time  Reference  Distribution  technique  de¬ 
scribed  here  also  selects  the  highest  ranking  clock  as  master. 
Within  this  approach,  the  natural  hierarchy  determined  by  the 
network  connectivity  is  employed;  instead  of  only  supplying 
timing  information  over  the  best  path  (of  least  total  demerit) 
to  each  node,  the  nodes  determine  the  error  in  their  local 
clocks  by  either  using  information  from  all  neighboring 
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(directly  connected)  nodes  higher  in  the  hierarchy  than  the 
local  node  or  by  using  information  from  all  neighboring  nodes 
not  lower  in  the  hierarchy.  To  do  this,  information  is 
exchanged  between  neighboring  nodes  and  there  are  a  set  of 
procedures  or  rules  for  applying  this  information.  Furthermore 
the  selection  of  the  paths  for  distribution  of  the  time  refer¬ 
ence  and  the  measurement  of  the  error  in  the  local  clock  are 
independent  of  the  correction  of  the  errors  in  any  of  the  other 
nodal  clocks.  This  permits  error  correction  in  any  clock  to  be 
made  with  minimum  perturbations  of  the  network  while  still  not 
interfering  with  the  accurate  measurement  of  the  error  in  any 
other  clock.  This  is  accomplished  by  having  each  node  in  the 
network  inform  its  neighbors  of  the  measured  but  uncorrected 
error  in  its  own  clock  [4] . 

The  Improved  Time  Reference  Distribution  technique  as 
defined  in  [4]  is  implemented  in  software.  In  lieu  of  pre¬ 
senting  the  entire  discussion,  the  following  paragraphs  are 
excerpts  from  [4]  and  should  provide  a  basic  understanding 
of  this  algorithm. 

Quite  a  large  number  of  desirable  characteristics  can  be 
listed  for  the  timing  subsystem  of  a  large  digital  communi¬ 
cations  network.  Although  it  cannot  be  proved  that  these 
characteristics  are  necessary,  many  of  them  are  widely 
accepted  as  desirable.  These  include:  (1)  Any  node 
should  be  able  to  obtain  all  required  timing  informa¬ 
tion  from  its  neighbors  (directly  connected  nodes) 
without  need  to  communicate  with  more  distant  nodes; 

(2)  The  timing  subsystem  should  accommodate  failures 
or  destruction  of  major  parts  of  the  network  and  still 
remain  operational;  (3)  Timing  perturbations  at  a  node 
should  not  propagate  to  other  parts  of  the  network 
(i.e.,  when  one  node  makes  a  correction  or  other  change 
in  its  clock,  it  is  not  desirable  for  this  to  propagate 
through  the  network  like  falling  dominoes);  (4)  There 
should  be  no  closed  timing  loop  that  could  potentially 
contribute  to  system  instability;  (5)  The  timing  subsystem 
should  permit  systematic  self-monitoring  to  provide  early 
detection  of  malfunctions  so  that  they  can  be  corrected 
before  they  interrupt  communications  traffic;  (6)  The 
communications  timing  subsystem  should  be  compatible 
with  other  timing  subsystems  such  as  those  employed  for 
navigation;  and  (7)  The  timing  subsystem  should  be 
self-organizing,  initially  and  following  failures. 
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An  error  in  a  clock  at  any  level  of  the  timing  hierarchy 
should  not  affect  the  measurement  of  the  clock  error  at 
any  node  lower  in  the  hierarchy.  Further,  the  selection 
of  the  paths  for  distribution  of  the  time  reference  and 
the  measurement  of  the  error  in  the  local  clock  should 
be  independent  of  the  correction  of  the  errors  in  any 
of  the  other  nodal  clocks.  This  permits  error  correction 
in  any  clock  to  be  made  with  minimum  perturbations  of  the 
network  while  still  not  interfering  with  the  accurate 
measurement  of  the  error  in  any  other  clock.  This  can  be 
accomplished  very  simply  by  having  each  node  in  the  net¬ 
work  inform  its  neighbors  of  the  measured  but  uncorrected 
error  in  its  own  clock. 

Two  classes  of  timing  information  are  maintained  at  each 
node : 


Class  1  Timing  information  is  based  only  on  infor¬ 
mation  received  from  nodes  higher  in  the 
hierarchy  than  the  local  node. 

Class  2  Timing  information  is  based  on  information 
received  from  all  nodes  not  lower  in  the 
hierarchy. 

In  order  to  provide  an  effective  self-organizing  method 
of  accurately  distributing  a  time  reference  through  a 
digital  communications  network,  information  is  exchanged 
between  neighboring  nodes.  This  information  is  applied 
in  compliance  with  a  set  of  rules.  These  rules  are  dis¬ 
cussed  in  the  text  after  the  following  list  of  informa¬ 
tion  which  is  transmitted  by  each  node  to  its  neighbors : 

INFO  1.  Rank  of  the  clock  used  as  the  master  time  refer*- 
ence  for  the  local  clock.  (This  information  is 
used  to  assure  that  the  highest  ranking  clock  in 
the  network  is  used  as  master  and  to  establish 
the  order  of  precedence  to  master  when  a  master 
fails.) 

INFO  2.  Number  of  links  between  the  local  node  and  its 
master  time  reference.  (This  information  is 
used  to  establish  the  desired  hierarchy.) 
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INFO  3. 


Time  of  the  clock  at  the  remote  end  of  the  link 
(Including  the  effect  of  the  time  required  for 
the  signal  to  transit  from  the  remote  node  to 
the  local  node)  as  measured  by  the  clock  at  the 
local  node. 

INFO  4A.  Measured  but  uncorrected  error  in  the  local 

clock  based  on  information  from  those  neighbors 
higher  in  the  timing  hierarchy  than  the  local 
node.  (The  term  measured  error  as  used  here 
includes  errors  obtained  by  mathematically  com¬ 
bining  other  measurements  and  the  error  in  this 
measured  error  will  be  called  its  inaccuracy.) 

INFO  4B.  Same  as  4A,  except  based  on  information  from 

those  neighbors  not  lower  in  the  hierarchy  than 
the  local  node. 

INFO  5A.  Estimated  inaccuracy,  stated  as  a  variance  (or 
standard  deviation) ,  of  the  local  clock  based 
on  information  from  all  neighboring  nodes 
higher  the  hierarchy  than  the  local  node. 

INFO  5B.  Same  as  5A,  except  based  on  information  from 

all  neighbors  not  lower  in  the  timing  hierarchy 
than  the  local  node. 

Notice  that  the  two  classes  of  information  under  item  4 
and  the  two  classes  of  information  under  item  5  are  dis¬ 
tinguished  by  the  sources  of  information  used  to  obtain 
them  and  also  by  the  nodes  that  make  use  of  them. 

When  the  rules  for  their  use  are  also  considered,  it 
will  be  observed  that  they  prevent  the  formation  of  the 
closed  feedback  paths. 

Each  node  applies  the  following  set  of  rules  for  the  use 
of  the  information  received  from  its  neighbors. 

RULE  1  A  node  initially  entering  the  network  will 
use  its  own  clock  as  its  time  reference  until  a  better 
reference  can  be  determined.  Its  own  clock  provides  a 
basic  time  reference  to  which  the  node  always  returns  when 
it  has  no  better  reference  available.  Under  these  condi¬ 
tions,  the  local  node  supplies  the  rank  of  its  own 
clock  to  its  neighbors  as  INFO  1. 
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RULE  2  The  first  type  of  information  received  froa 
neighboring  nodes,  INFO  1,  provides  the  local  node  with 
the  rank  of  the  clock  used  as  the  master  time  reference 
by  each  of  its  neighbors.  If  one  or  more  neighbors' 
reference  clocks  outrank  the  local  clock,  the  node  will 
select  those  neighbors  (or  the  single  neighbor)  referenc¬ 
ing  the  highest  ranking  master  and  use  them  (or  it)  in  de¬ 
termining  its  own  time  reference,  i.e.,  measuring  the 
error  in  its  own  clock.  The  rank  of  the  master  time  refer 
ence  used  by  the  selected  neighbors  will  be  supplied  to 
all  neighbors  as  INFO  1,  i.e.,  the  rank  of  the  clock  used 
as  master  for  the  local  node.  Continued  application  of 
this  rule  by  all  nodes  will  result  in  all  nodes  refer¬ 
encing  the  same  highest  ranking  master  clock. 

RULE  3  If  the  local  node  is  referencing  its  own  clock 
there  are  no  links  between  the  local  node  and  its  master 
reference  and  this  information  is  supplied  to  its  neigh¬ 
bors  as  INFO  2.  The  second  type  of  information,  INFO  2, 
as  received  from  its  neighbors  provides  the  local  node 
with  information  about  the  number  of  links  between 
each  neighboring  node  and  that  neighbor's  master  time 
reference.  Unless  the  clock  at  the  local  node  outranks 
the  master  reference  of  all  of  its  neighbors,  the  number 
of  links  between  the  local  node  and  the  master  is  greater 
by  one  than  that  of  the  neighbors  (or  neighbor)  selected 
by  Rule  2  which  have  the  least  number  of  links  between 
themselves  and  their  master.  This  information  is 
supplied  to  the  neighboring  nodes  as  INFO  2.  Continued 
application  of  this  rule  will  result  in  establishing  the 
desired  natural  hierarchy.  INFO  2  information  as  trans¬ 
mitted  to  neighboring  nodes  and  as  received  from  them 
indicates  the  position  in  the  hierarchy  of  the  local  node 
relative  to  each  of  its  neighbors. 

RULE  4  The  third  type  of  information,  INFO  3,  as 
received  from  neighboring  nodes  provides  the  local  node 
with  the  time  difference  between  the  local  clock  and  the 
clock  at  each  neighboring  node  (including  the  signal 
transit  time  from  the  local  node  to  the  neighboring  node.) 
INFO  3  as  transmitted  to  the  corresponding  neighboring 
node  is  subtracted  from  this  information,  and  the  dif¬ 
ference  is  divided  by  2.  This  provides  a  measurement  of 
the  actual  time  difference  (no  transit  time  included) 
between  the  local  clock  and  the  clock  at  each  neighboring 
node. 


A-10 


RULE  5  Each  neighboring  node  not  higher  in  the  hier¬ 
archy  than  the  local  node  transmits  to  the  local  node,  as 
INFO  4A,  the  Class  1  measured  but  uncorrected  error  of  its 
own  clock,  i.e.,  the  error  determined  using  information 
from  that  neighbor's  neighbors  that  are  higher  in  the  hier¬ 
archy  than  the  neighbor.  Similarly,  each  neighbor  higher 
in  the  hierarchy  than  the  local  node  transmits  to  the  local 
node,  as  INFO  4B,  the  Class  2  measured  but  uncorrected 
error  in  its  own  clock,  i.e.,  the  error  determined  using 
information  from  that  neighbor's  neighbors  that  are  not 
lower  in  the  hierarchy  than  the  neighbor.  As  received,  thi 
this  information  gives  a  measured  but  uncorrected  error  for 
each  neighboring  node.  To  this  is  added  the  difference 
between  the  local  clock  and  each  neighboring  clock  as  deter¬ 
mined  by  Rule  4. 

The  result  is  a  set  of  error  measurements  for  the  local 
clock  based  on  information  from  each  of  its  neighbors. 

(The  reason  for  using  Class  1  information  from  some  neigh¬ 
bors  and  Class  2  information  from  others  is  to  avoid 
closed  feedback  paths  while  still  making  very  effective  use 
of  the  available  information.) 

Figures  2,  3  and  4  depict  the  application  of  TRD  rules  1-5 
including  an  obsolete  data  handler  similar  to  the  previous  TRD 
version  discussed  in  the  first  section  of  this  Appendix. 

The  following  rules  are  represented  by  flow  diagrams  in 
Figure  5.  Each  portion  delineated  by  the  rules  described  below. 

RULE  6  Each  neighboring  node  not  higher  in  the  hier¬ 
archy  than  the  local  node  transmits  to  the  local  node, 
as  INFO  5A,  the  estimated  inaccuracy,  stated  as  a  vari¬ 
ance  (or  standard  deviation),  of  its  Class  1  measured 
error.  Similarly,  each  neighboring  node  higher  in  the 
hierarchy  than  the  local  node  transmits  to  the  local 
node,  as  INFO  5B,  the  estimated  inaccuracy,  stated  as  a 
variance  (or  standard  deviation) ,  of  its  Class  2  measured 
error.  This  information,  as  received,  is  the  estimated 
inaccuracy  of  the  measured  but  uncorrected  error  associated 
with  each  neighboring  node.  Add  to  each  member  of  this 
set  of  information  (directly  if  stated  as  variances  or 
as  the  square  root  of  the  sum  of  the  squares  if  stated 
as  standard  deviations)  the  estimated  inaccuracy  of  the 


( 

; 


NOTE: 


TRDNEW :  HIERARCHY  DETERMINATION 


RULE  1 


NODES  JUST  ENTER¬ 
ING  THE  NETWORK 
WILL  REFERENCE  THEIR 
OWN  CLOCKS  UNTIL  A 
BETTER  REFERENCE  IS 
DETERMINED. 


MORE  \ 
LINKS  . 

>  TMPLNG 

.  ? 

[NO 

LENGTH 

$ 

RANK 

REFLOC 

REFRNK 

LNKLTH 

TEMPORARY  SHORTEST 
DIST  TO  MASTER 
DIST;  LOCAL  NODE  TO 
CURRENT  MASTER 
LOCAL  NODE  RANK 
RANK  OF  LOCAL  NODES 
ULT.  REF 

SELECTED  NEIGHBOR 
REFERENCING  HIGHEST 
RANKING  CLOCK 
DIST;  SELECTED 
NEIGHBOR  TO  ITS 
CURRENT  MASTER 


Figure  2  TRD  Hierarchy  Determination 


A-12 


TRD:  OBSOLETE  DATA  HANDLER 


RULE  D1 


RULE  D2 


RULE  D3 


Figure  3  TRD  Obsolete  Data  Handler 
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link  between  each  neighbor  and  the  local  node  as  deter¬ 
mined  during  engineering  design.  The  result  is  a  set  of 
inaccuracies  for  the  set  of  measured  errors  in  the  local 
clock  based  on  information  from  each  neighbor. 

The  estimated  inaccuracy  attributed  to  the  link  between  the 
local  node  and  a  neighbor  as  established  during  engineering 
design  includes  several  parameters.  It  includes  an  effect 
due  to  the  differences  in  signal  transit  time  in  the  two 
directions  of  the  duplex  link  which  includes  delay  differ¬ 
ences  in  the  transmitters  and  receivers  at  the  two  ends  of 
the  link.  It  also  includes  inaccuracies  in  the  equipment 
used  to  measure  timing  differences  between  the  received  sig¬ 
nal  and  the  local  clock. 

RULE  7  From  the  set  of  error  measurements  for  the 
local  clock  as  determined  by  Rule  5,  and  the  associated 
inaccuracies  determined  by  Rule  6,  only  those  for  neigh¬ 
bors  higher  in  the  timing  hierarchy  than  the  local  node 
are  selected.  These  error  measurements  are  combined  to 
determine  a  Class  1  measurement  of  the  error  in  the 
local  clock,  i.e.,  one  based  on  neighbors  higher  in  the 
hierarchy  than  the  local  node.  This  is  supplied  as  INFO 
4A,  the  measured  error  in  the  local  clock,  to  all  neigh¬ 
bors  not  lower  in  the  timing  hierarchy  than  the  local 
clock. 

RULE  8  From  the  set  of  inaccuracies  determined  by 
Rule  6  only  those  for  neighboring  nodes  higher  in  the 
hierarchy  than  the  local  node  are  selected.  These  are 
combined  to  determine  the  inaccuracy  for  the  measured 
error  in  the  local  clock  based  on  information  from  neigh¬ 
bors  higher  in  the  hierarchy  than  the  local  node. 

This  information  is  supplied  as  INFO  5A  to  all  neighbors 
not  lower  in  the  timing  hierarchy  than  the  local  node. 

RULE  9  From  the  set  of  error  measurements  for  the 
local  clock  as  determined  by  Rule  5  and  the  associated 
inaccuracies  determined  from  Rule  6,  all  those  for 
neighbors  not  lower  in  the  hierarchy  than  the  local  node 
are  selected.  These  error  measurements  are  combined  to 
determine  a  Class  2  measurement  of  the  error  in  the  local 
clock,  i.e.,  one  based  on  all  those  neighbors  not  lower 
in  the  timing  hierarchy  than  the  local  node.  This  is 
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supplied  as  INFO  4B,  the  measured  error  in  the  local  clock, 
to  all  neighbors  lower  in  the  timing  hierarchy  than  the 
local  clock. 

RULE  10  From  the  set  of  inaccuracies  determined  by 
Rule  6  all  those  for  neighbors  not  lower  in  the  timing 
hierarchy  than  the  local  node  are  selected.  These 
inaccuracies  are  combined  to  determine  the  inaccuracy  for 
the  measured  error  in  the  local  clock  based  on  informa¬ 
tion  from  all  those  neighbors  not  lower  in  the  timing 
hierarchy  than  the  local  node.  This  inaccuracy  information 
is  provided  as  INFO  5B  to  all  neighbors  lower  in  the 
timing  hierarchy  than  the  local  node. 

The  combining  of  information  over  several  different  paths, 
in  addition  to  providing  more  accurate  time  measurements 
at  many  nodes  remote  from  the  master,  reducing  the  need 
for  massive  reorganization  of  the  network  following 
some  failures  as  required  when  using  only  the. best  path, 
also  provides  the  possibility  for  quantitative  evaluation 
of  the  fitness  of  the  timing  subsystem.  Since  each  time 
error  measurement  (the  term  measurement  as  used  here 
includes  the  mathematical  combination  of  measurement 
information  from  different  sources)  has  a  corresponding 
estimated  inaccuracy,  these  time  error  measurements  and 
their  corresponding  inaccuracy  estimates  can  be  used  to 
provide  a  quantitative  alarm  system.  This  leads  to 
Rule  11. 

RULE  11  Rule  5  provides  a  set  of  error  measurements 
for  the  local  clock  based  on  information  from  each  neigh¬ 
boring  node.  Rule  6  provides  a  corresponding  set  of 
inaccuracies  for  these  error  measurements.  Rule  9  pro¬ 
vides  a  combined  measurement  for  the  error  in  the  local 
clock.  Rule  10  provides  a  corresponding  inaccuracy  for 
the  combined  measurement.  The  combined  measurement  as 
determined  by  Rule  9  is  subtracted  from  each  member  of  the 
set  of  error  measurements  determined  by  Rule  5.  The 
resulting  set  gives  the  difference  between  each  individual 
measurement  and  the  combined  measurement.  The  inaccu¬ 
racy  (stated  as  a  variance)  determined  by  Rule  10  is  added 
to  each  member  of  the  set  of  inaccuracies  obtained  by 
Rule  6  (also  stated  as  a  variance)  and  the  square  root 
of  each  member  of  this  set  is  taken  to  obtain  a  set  of 
estimates  of  the  standard  deviations  of  the  clock  error 


measurements  based  on  information  from  each  neighbor  rela¬ 
tive  to  the  combined  clock  error  measurement.  Each  member 
of  the  set  of  differences  between  individual  measurements 
and  the  combined  measurement  is  divided  by  the  estimate  of 
the  corresponding  standard  deviation  to  obtain  a  normal¬ 
ized  set  of  ratios.  These  ratios  provide  levels  of  alarms 
as  the  value  climbs  from  2  to  5.  A  level  5  alarm  has  a 
chance  of  one  in  a  million  of  occurring  in  a  normally 
operating  system. 

The  alarm  system  is  described  in  the  flow  diagram  of 
Figure  6. 
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Figure  6  TRDNEW  Quantitative  Alarm  System 
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